Sep 11 12:12:20.658316 ip-10-0-131-172 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 12:12:20.658330 ip-10-0-131-172 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 12:12:20.658340 ip-10-0-131-172 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 12:12:20.658547 ip-10-0-131-172 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 12:12:31.835799 ip-10-0-131-172 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 12:12:31.835814 ip-10-0-131-172 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 24f67bfd44b749d8814b18d28132b1b5 --
Sep 11 12:14:27.293527 ip-10-0-131-172 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 12:14:27.685522 ip-10-0-131-172 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 12:14:27.685522 ip-10-0-131-172 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 12:14:27.685522 ip-10-0-131-172 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 12:14:27.685522 ip-10-0-131-172 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 12:14:27.685522 ip-10-0-131-172 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 12:14:27.686526 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.686445    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 12:14:27.689362 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689348    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 12:14:27.689362 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689361    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 12:14:27.689423 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689365    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 12:14:27.689423 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689369    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 12:14:27.689423 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689372    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 12:14:27.689423 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689377    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 12:14:27.689423 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689381    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 12:14:27.689423 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689389    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 12:14:27.689423 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689393    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 12:14:27.689423 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689396    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 12:14:27.689423 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689398    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 12:14:27.689423 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689401    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 12:14:27.689423 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689403    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 12:14:27.689423 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689406    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 12:14:27.689423 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689409    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 12:14:27.689423 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689411    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 12:14:27.689423 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689414    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 12:14:27.689423 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689417    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 12:14:27.689423 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689419    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 12:14:27.689423 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689430    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 12:14:27.689881 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689433    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 12:14:27.689881 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689436    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 12:14:27.689881 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689439    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 12:14:27.689881 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689442    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 12:14:27.689881 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689444    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 12:14:27.689881 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689448    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 12:14:27.689881 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689450    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 12:14:27.689881 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689453    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 12:14:27.689881 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689455    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 12:14:27.689881 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689458    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 12:14:27.689881 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689461    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 12:14:27.689881 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689463    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 12:14:27.689881 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689466    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 12:14:27.689881 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689468    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 12:14:27.689881 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689471    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 12:14:27.689881 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689473    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 12:14:27.689881 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689476    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 12:14:27.689881 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689479    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 12:14:27.689881 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689482    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 12:14:27.689881 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689484    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 12:14:27.690378 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689487    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 12:14:27.690378 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689489    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 12:14:27.690378 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689493    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 12:14:27.690378 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689496    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 12:14:27.690378 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689499    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 12:14:27.690378 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689502    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 12:14:27.690378 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689505    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 12:14:27.690378 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689508    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 12:14:27.690378 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689512    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 12:14:27.690378 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689516    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 12:14:27.690378 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689519    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 12:14:27.690378 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689522    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 12:14:27.690378 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689525    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 12:14:27.690378 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689528    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 12:14:27.690378 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689531    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 12:14:27.690378 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689534    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 12:14:27.690378 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689536    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 12:14:27.690378 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689539    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 12:14:27.690378 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689541    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 12:14:27.690838 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689544    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 12:14:27.690838 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689547    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 12:14:27.690838 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689549    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 12:14:27.690838 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689552    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 12:14:27.690838 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689554    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 12:14:27.690838 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689557    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 12:14:27.690838 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689559    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 12:14:27.690838 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689562    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 12:14:27.690838 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689564    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 12:14:27.690838 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689567    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 12:14:27.690838 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689570    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 12:14:27.690838 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689573    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 12:14:27.690838 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689576    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 12:14:27.690838 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689579    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 12:14:27.690838 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689581    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 12:14:27.690838 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689584    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 12:14:27.690838 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689586    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 12:14:27.690838 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689589    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 12:14:27.690838 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689592    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 12:14:27.691310 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689595    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 12:14:27.691310 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689597    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 12:14:27.691310 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689600    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 12:14:27.691310 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689602    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 12:14:27.691310 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689605    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 12:14:27.691310 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689607    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 12:14:27.691310 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689611    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 12:14:27.691310 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689615    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 12:14:27.691310 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689618    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 12:14:27.691310 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689621    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 12:14:27.691310 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689623    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 12:14:27.691310 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689626    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 12:14:27.691310 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689628    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 12:14:27.691310 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689631    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 12:14:27.691310 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689633    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 12:14:27.691310 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689636    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 12:14:27.691310 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689639    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 12:14:27.691310 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689641    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 12:14:27.691310 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689644    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 12:14:27.691310 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689647    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 12:14:27.691789 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.689649    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 12:14:27.691789 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690345    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 12:14:27.691789 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690352    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 12:14:27.691789 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690356    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 12:14:27.691789 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690359    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 12:14:27.691789 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690362    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 12:14:27.691789 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690366    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 12:14:27.691789 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690369    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 12:14:27.691789 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690372    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 12:14:27.691789 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690375    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 12:14:27.691789 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690377    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 12:14:27.691789 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690380    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 12:14:27.691789 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690383    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 12:14:27.691789 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690386    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 12:14:27.691789 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690389    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 12:14:27.691789 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690391    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 12:14:27.691789 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690394    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 12:14:27.691789 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690397    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 12:14:27.691789 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690399    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 12:14:27.691789 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690402    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 12:14:27.692337 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690405    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 12:14:27.692337 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690408    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 12:14:27.692337 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690410    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 12:14:27.692337 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690413    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 12:14:27.692337 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690416    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 12:14:27.692337 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690418    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 12:14:27.692337 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690421    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 12:14:27.692337 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690425    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 12:14:27.692337 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690429    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 12:14:27.692337 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690432    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 12:14:27.692337 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690435    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 12:14:27.692337 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690439    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 12:14:27.692337 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690442    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 12:14:27.692337 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690445    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 12:14:27.692337 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690448    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 12:14:27.692337 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690452    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 12:14:27.692337 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690454    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 12:14:27.692337 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690457    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 12:14:27.692337 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690459    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 12:14:27.692806 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690462    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 12:14:27.692806 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690465    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 12:14:27.692806 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690468    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 12:14:27.692806 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690471    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 12:14:27.692806 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690474    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 12:14:27.692806 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690477    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 12:14:27.692806 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690480    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 12:14:27.692806 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690483    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 12:14:27.692806 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690486    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 12:14:27.692806 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690488    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 12:14:27.692806 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690491    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 12:14:27.692806 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690493    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 12:14:27.692806 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690496    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 12:14:27.692806 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690499    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 12:14:27.692806 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690502    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 12:14:27.692806 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690505    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 12:14:27.692806 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690507    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 12:14:27.692806 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690509    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 12:14:27.692806 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690512    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 12:14:27.692806 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690514    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 12:14:27.693314 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690517    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 12:14:27.693314 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690519    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 12:14:27.693314 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690522    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 12:14:27.693314 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690524    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 12:14:27.693314 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690527    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 12:14:27.693314 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690529    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 12:14:27.693314 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690532    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 12:14:27.693314 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690534    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 12:14:27.693314 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690536    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 12:14:27.693314 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690539    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 12:14:27.693314 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690541    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 12:14:27.693314 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690544    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 12:14:27.693314 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690546    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 12:14:27.693314 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690549    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 12:14:27.693314 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690552    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 12:14:27.693314 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690555    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 12:14:27.693314 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690558    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 12:14:27.693314 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690560    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 12:14:27.693314 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690564    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 12:14:27.693314 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690566    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 12:14:27.693801 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690568    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 12:14:27.693801 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690571    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 12:14:27.693801 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690574    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 12:14:27.693801 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690576    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 12:14:27.693801 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690579    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 12:14:27.693801 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690582    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 12:14:27.693801 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690584    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 12:14:27.693801 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690587    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 12:14:27.693801 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690589    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 12:14:27.693801 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690592    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 12:14:27.693801 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690594    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 12:14:27.693801 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690596    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 12:14:27.693801 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690599    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 12:14:27.693801 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690601    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 12:14:27.693801 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690604    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 12:14:27.693801 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690606    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 12:14:27.693801 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690609    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 12:14:27.693801 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690612    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 12:14:27.693801 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690614    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 12:14:27.693801 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690617    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 12:14:27.694298 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690619    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 12:14:27.694298 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690979    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 12:14:27.694298 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690984    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 12:14:27.694298 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690987    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 12:14:27.694298 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690990    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 12:14:27.694298 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690993    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 12:14:27.694298 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.690996    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 12:14:27.694298 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691001    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 12:14:27.694298 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691004    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 12:14:27.694298 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691007    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 12:14:27.694298 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691010    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 12:14:27.694298 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691012    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 12:14:27.694298 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691015    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 12:14:27.694298 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691018    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 12:14:27.694298 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691020    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 12:14:27.694298 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691023    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 12:14:27.694298 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691027    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 12:14:27.694298 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691030    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 12:14:27.694298 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691032    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 12:14:27.694298 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691035    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 12:14:27.694813 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691038    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 12:14:27.694813 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691041    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 12:14:27.694813 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691043    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 12:14:27.694813 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691046    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 12:14:27.694813 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691048    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 12:14:27.694813 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691051    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 12:14:27.694813 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691055    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 12:14:27.694813 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691058    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 12:14:27.694813 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691061    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 12:14:27.694813 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691064    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 12:14:27.694813 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691066    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 12:14:27.694813 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691069    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 12:14:27.694813 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691071    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 12:14:27.694813 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691074    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 12:14:27.694813 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691076    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 12:14:27.694813 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691079    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 12:14:27.694813 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691081    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 12:14:27.694813 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691084    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 12:14:27.694813 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691087    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 12:14:27.695299 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691090    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 12:14:27.695299 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691093    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 12:14:27.695299 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691096    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 12:14:27.695299 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691098    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 12:14:27.695299 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691101    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 12:14:27.695299 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691103    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 12:14:27.695299 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691120    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 12:14:27.695299 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691123    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 12:14:27.695299 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691126    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 12:14:27.695299 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691131    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 12:14:27.695299 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691133    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 12:14:27.695299 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691136    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 12:14:27.695299 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691138    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 12:14:27.695299 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691141    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 12:14:27.695299 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691144    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 12:14:27.695299 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691146    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 12:14:27.695299 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691149    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 12:14:27.695299 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691151    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 12:14:27.695299 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691154    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 12:14:27.695299 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691156    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 12:14:27.695777 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691166    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 12:14:27.695777 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691170    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 12:14:27.695777 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691172    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 12:14:27.695777 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691175    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 12:14:27.695777 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691180    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 12:14:27.695777 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691184    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 12:14:27.695777 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691187    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 12:14:27.695777 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691190    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 12:14:27.695777 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691193    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 12:14:27.695777 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691195    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 12:14:27.695777 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691198    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 12:14:27.695777 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691200    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 12:14:27.695777 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691206    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 12:14:27.695777 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691209    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 12:14:27.695777 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691211    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 12:14:27.695777 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691214    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 12:14:27.695777 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691216    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 12:14:27.695777 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691220    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 12:14:27.695777 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691223    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 12:14:27.696251 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691226    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 12:14:27.696251 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691229    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 12:14:27.696251 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691233    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 12:14:27.696251 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691236    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 12:14:27.696251 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691238    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 12:14:27.696251 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691242    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 12:14:27.696251 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691244    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 12:14:27.696251 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691247    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 12:14:27.696251 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691250    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 12:14:27.696251 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691252    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 12:14:27.696251 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691255    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 12:14:27.696251 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691258    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 12:14:27.696251 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691261    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 12:14:27.696251 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691263    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 12:14:27.696251 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691266    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 12:14:27.696251 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691268    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 12:14:27.696251 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691271    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 12:14:27.696251 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691274    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 12:14:27.696251 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691276    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 12:14:27.696251 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691279    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 12:14:27.696733 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691282    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 12:14:27.696733 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.691284    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 12:14:27.696733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.691355    2561 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 12:14:27.696733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.691402    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 12:14:27.696733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.691408    2561 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 12:14:27.696733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.691414    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 12:14:27.696733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.691419    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 12:14:27.696733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.691422    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 12:14:27.696733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.691426    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 12:14:27.696733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.691430    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 12:14:27.696733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.691433    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 12:14:27.696733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.691436    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 12:14:27.696733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.691439    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 12:14:27.696733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.691442    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 12:14:27.696733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.691447    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 12:14:27.696733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.691450    2561 flags.go:64] FLAG: --cgroup-root=""
Sep 11 12:14:27.696733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.691453    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 12:14:27.696733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.691456    2561 flags.go:64] FLAG: --client-ca-file=""
Sep 11 12:14:27.696733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.691459    2561 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 12:14:27.696733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.691462    2561 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 12:14:27.696733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692166    2561 flags.go:64] FLAG: --cluster-domain=""
Sep 11 12:14:27.696733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692170    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 12:14:27.696733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692173    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 12:14:27.697295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692177    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 12:14:27.697295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692180    2561 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 12:14:27.697295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692184    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 12:14:27.697295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692187    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 12:14:27.697295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692190    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 12:14:27.697295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692194    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 12:14:27.697295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692198    2561 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 12:14:27.697295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692201    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 12:14:27.697295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692203    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 12:14:27.697295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692207    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 12:14:27.697295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692210    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 12:14:27.697295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692213    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 12:14:27.697295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692217    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 12:14:27.697295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692220    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 12:14:27.697295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692223    2561 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 12:14:27.697295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692227    2561 flags.go:64] FLAG: --enable-server="true"
Sep 11 12:14:27.697295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692230    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 12:14:27.697295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692239    2561 flags.go:64] FLAG: --event-burst="100"
Sep 11 12:14:27.697295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692242    2561 flags.go:64] FLAG: --event-qps="50"
Sep 11 12:14:27.697295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692245    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 12:14:27.697295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692248    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 12:14:27.697295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692251    2561 flags.go:64] FLAG: --eviction-hard=""
Sep 11 12:14:27.697295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692254    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 12:14:27.697295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692259    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 12:14:27.697959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692262    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 12:14:27.697959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692265    2561 flags.go:64] FLAG: --eviction-soft=""
Sep 11 12:14:27.697959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692268    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 12:14:27.697959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692271    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 12:14:27.697959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692274    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 12:14:27.697959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692276    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 12:14:27.697959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692279    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 12:14:27.697959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692282    2561 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 12:14:27.697959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692285    2561 flags.go:64] FLAG: --feature-gates=""
Sep 11 12:14:27.697959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692289    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 12:14:27.697959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692292    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 12:14:27.697959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692295    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 12:14:27.697959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692298    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 12:14:27.697959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692301    2561 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 12:14:27.697959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692304    2561 flags.go:64] FLAG: --help="false"
Sep 11 12:14:27.697959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692307    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-131-172.ec2.internal"
Sep 11 12:14:27.697959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692310    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 12:14:27.697959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692313    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 12:14:27.697959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692316    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 12:14:27.697959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692319    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 12:14:27.697959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692322    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 12:14:27.697959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692325    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 12:14:27.697959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692328    2561 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692331    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692334    2561 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692337    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692341    2561 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692344    2561 flags.go:64] FLAG: --kube-reserved=""
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692347    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692350    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692353    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692357    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692360    2561 flags.go:64] FLAG: --lock-file=""
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692363    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692365    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692368    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692378    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692381    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692384    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692387    2561 flags.go:64] FLAG: --logging-format="text"
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692390    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692393    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692396    2561 flags.go:64] FLAG: --manifest-url=""
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692399    2561 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692403    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692406    2561 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692410    2561 flags.go:64] FLAG: --max-pods="110"
Sep 11 12:14:27.698703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692413    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 12:14:27.699763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692416    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 12:14:27.699763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692419    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 12:14:27.699763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692422    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 12:14:27.699763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692425    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 12:14:27.699763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692428    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 12:14:27.699763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692431    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 12:14:27.699763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692438    2561 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 12:14:27.699763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692441    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 12:14:27.699763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692444    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 12:14:27.699763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692447    2561 flags.go:64] FLAG: --pod-cidr=""
Sep 11 12:14:27.699763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692451    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 12:14:27.699763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692456    2561 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 12:14:27.699763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692459    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 12:14:27.699763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692462    2561 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 12:14:27.699763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692465    2561 flags.go:64] FLAG: --port="10250"
Sep 11 12:14:27.699763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692468    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 12:14:27.699763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692472    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0091309fb0e2b7c94"
Sep 11 12:14:27.699763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692475    2561 flags.go:64] FLAG: --qos-reserved=""
Sep 11 12:14:27.699763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692478    2561 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 12:14:27.699763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692481    2561 flags.go:64] FLAG: --register-node="true"
Sep 11 12:14:27.699763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692484    2561 flags.go:64] FLAG: --register-with-taints=""
Sep 11 12:14:27.699763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692488    2561 flags.go:64] FLAG: --registry-burst="10"
Sep 11 12:14:27.699763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692491    2561 flags.go:64] FLAG: --registry-qps="5"
Sep 11 12:14:27.699763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692494    2561 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692496    2561 flags.go:64] FLAG: --reserved-memory=""
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692500    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692503    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692506    2561 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692509    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692512    2561 flags.go:64] FLAG: --runonce="false"
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692515    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692517    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692521    2561 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692523    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692526    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692529    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692532    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692535    2561 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692538    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692541    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692544    2561 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692546    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692549    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692553    2561 flags.go:64] FLAG: --system-cgroups=""
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692555    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692561    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692564    2561 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692567    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 12:14:27.700798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692572    2561 flags.go:64] FLAG: --tls-min-version=""
Sep 11 12:14:27.701902 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692575    2561 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 12:14:27.701902 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692578    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 12:14:27.701902 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692581    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 12:14:27.701902 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692584    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 12:14:27.701902 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692587    2561 flags.go:64] FLAG: --v="2"
Sep 11 12:14:27.701902 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692592    2561 flags.go:64] FLAG: --version="false"
Sep 11 12:14:27.701902 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692596    2561 flags.go:64] FLAG: --vmodule=""
Sep 11 12:14:27.701902 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692599    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 12:14:27.701902 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692603    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 12:14:27.701902 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692712    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 12:14:27.701902 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692716    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 12:14:27.701902 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692719    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 12:14:27.701902 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692722    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 12:14:27.701902 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692727    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 12:14:27.701902 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692730    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 12:14:27.701902 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692733    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 12:14:27.701902 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692736    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 12:14:27.701902 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692738    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 12:14:27.701902 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692741    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 12:14:27.701902 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692744    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 12:14:27.701902 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692747    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 12:14:27.702762 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692749    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 12:14:27.702762 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692752    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 12:14:27.702762 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692755    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 12:14:27.702762 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692757    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 12:14:27.702762 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692759    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 12:14:27.702762 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692762    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 12:14:27.702762 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692765    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 12:14:27.702762 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692768    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 12:14:27.702762 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692770    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 12:14:27.702762 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692773    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 12:14:27.702762 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692777    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 12:14:27.702762 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692780    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 12:14:27.702762 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692782    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 12:14:27.702762 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692785    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 12:14:27.702762 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692788    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 12:14:27.702762 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692790    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 12:14:27.702762 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692795    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 12:14:27.702762 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692798    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 12:14:27.702762 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692801    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 12:14:27.702762 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692804    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 12:14:27.703356 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692807    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 12:14:27.703356 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692810    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 12:14:27.703356 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692812    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 12:14:27.703356 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692815    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 12:14:27.703356 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692818    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 12:14:27.703356 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692821    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 12:14:27.703356 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692825    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 12:14:27.703356 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692828    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 12:14:27.703356 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692831    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 12:14:27.703356 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692833    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 12:14:27.703356 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692836    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 12:14:27.703356 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692839    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 12:14:27.703356 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692842    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 12:14:27.703356 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692845    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 12:14:27.703356 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692848    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 12:14:27.703356 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692850    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 12:14:27.703356 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692853    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 12:14:27.703356 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692856    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 12:14:27.703356 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692858    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 12:14:27.703833 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692861    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 12:14:27.703833 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692864    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 12:14:27.703833 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692867    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 12:14:27.703833 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692870    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 12:14:27.703833 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692873    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 12:14:27.703833 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692876    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 12:14:27.703833 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692878    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 12:14:27.703833 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692881    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 12:14:27.703833 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692884    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 12:14:27.703833 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692888    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 12:14:27.703833 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692891    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 12:14:27.703833 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692893    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 12:14:27.703833 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692896    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 12:14:27.703833 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692898    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 12:14:27.703833 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692901    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 12:14:27.703833 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692903    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 12:14:27.703833 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692906    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 12:14:27.703833 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692908    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 12:14:27.703833 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692911    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 12:14:27.703833 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692914    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 12:14:27.704358 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692916    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 12:14:27.704358 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692919    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 12:14:27.704358 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692921    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 12:14:27.704358 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692924    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 12:14:27.704358 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692926    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 12:14:27.704358 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692929    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 12:14:27.704358 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692931    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 12:14:27.704358 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692934    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 12:14:27.704358 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692937    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 12:14:27.704358 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692939    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 12:14:27.704358 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692943    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 12:14:27.704358 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692947    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 12:14:27.704358 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692950    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 12:14:27.704358 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692953    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 12:14:27.704358 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692955    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 12:14:27.704358 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692958    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 12:14:27.704358 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692961    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 12:14:27.704358 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692963    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 12:14:27.704358 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692965    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 12:14:27.704358 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692968    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 12:14:27.704853 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692970    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 12:14:27.704853 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692975    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 12:14:27.704853 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692977    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 12:14:27.704853 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692979    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 12:14:27.704853 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692982    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 12:14:27.704853 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692985    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 12:14:27.704853 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692987    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 12:14:27.704853 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.692990    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 12:14:27.704853 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.692995    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 12:14:27.704853 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.700586    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 12:14:27.704853 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.700605    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 12:14:27.704853 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.700627    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 12:14:27.704853 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.700638    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 12:14:27.704853 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700718    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 12:14:27.705234 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700729    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 12:14:27.705234 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700735    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 12:14:27.705234 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700741    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 12:14:27.705234 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700746    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 12:14:27.705234 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700751    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 12:14:27.705234 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700755    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 12:14:27.705234 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700761    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 12:14:27.705234 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700766    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 12:14:27.705234 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700771    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 12:14:27.705234 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700776    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 12:14:27.705234 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700780    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 12:14:27.705234 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700784    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 12:14:27.705234 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700789    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 12:14:27.705234 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700793    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 12:14:27.705234 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700797    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 12:14:27.705234 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700801    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 12:14:27.705234 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700805    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 12:14:27.705234 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700810    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 12:14:27.705234 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700814    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 12:14:27.705234 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700818    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 12:14:27.705724 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700822    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 12:14:27.705724 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700827    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 12:14:27.705724 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700831    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 12:14:27.705724 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700836    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 12:14:27.705724 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700840    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 12:14:27.705724 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700844    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 12:14:27.705724 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700849    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 12:14:27.705724 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700853    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 12:14:27.705724 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700860    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 12:14:27.705724 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700864    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 12:14:27.705724 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700868    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 12:14:27.705724 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700872    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 12:14:27.705724 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700876    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 12:14:27.705724 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700881    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 12:14:27.705724 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700885    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 12:14:27.705724 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700889    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 12:14:27.705724 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700893    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 12:14:27.705724 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700898    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 12:14:27.705724 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700902    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 12:14:27.705724 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700906    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 12:14:27.706232 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700910    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 12:14:27.706232 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700914    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 12:14:27.706232 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700919    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 12:14:27.706232 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700923    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 12:14:27.706232 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700928    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 12:14:27.706232 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700932    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 12:14:27.706232 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700936    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 12:14:27.706232 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700940    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 12:14:27.706232 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700944    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 12:14:27.706232 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700948    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 12:14:27.706232 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700952    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 12:14:27.706232 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700957    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 12:14:27.706232 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700961    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 12:14:27.706232 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700965    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 12:14:27.706232 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700970    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 12:14:27.706232 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700974    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 12:14:27.706232 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700978    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 12:14:27.706232 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700982    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 12:14:27.706232 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700988    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 12:14:27.706720 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.700994    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 12:14:27.706720 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701000    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 12:14:27.706720 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701005    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 12:14:27.706720 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701009    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 12:14:27.706720 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701013    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 12:14:27.706720 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701017    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 12:14:27.706720 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701021    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 12:14:27.706720 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701025    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 12:14:27.706720 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701029    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 12:14:27.706720 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701033    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 12:14:27.706720 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701037    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 12:14:27.706720 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701041    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 12:14:27.706720 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701045    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 12:14:27.706720 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701049    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 12:14:27.706720 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701053    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 12:14:27.706720 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701057    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 12:14:27.706720 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701062    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 12:14:27.706720 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701066    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 12:14:27.706720 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701070    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 12:14:27.706720 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701074    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 12:14:27.707240 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701078    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 12:14:27.707240 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701082    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 12:14:27.707240 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701089    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 12:14:27.707240 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701095    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 12:14:27.707240 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701099    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 12:14:27.707240 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701119    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 12:14:27.707240 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701125    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 12:14:27.707240 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701130    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 12:14:27.707240 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701134    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 12:14:27.707240 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701137    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 12:14:27.707240 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701142    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 12:14:27.707240 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701146    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 12:14:27.707240 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701151    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 12:14:27.707240 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701155    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 12:14:27.707240 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701160    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 12:14:27.707240 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701165    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 12:14:27.707240 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701169    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 12:14:27.707240 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701173    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 12:14:27.707240 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701177    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 12:14:27.707703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.701186    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 12:14:27.707703 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701357    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 12:14:27.707703 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701365    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 12:14:27.707703 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701370    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 12:14:27.707703 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701375    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 12:14:27.707703 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701379    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 12:14:27.707703 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701383    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 12:14:27.707703 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701387    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 12:14:27.707703 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701391    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 12:14:27.707703 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701396    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 12:14:27.707703 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701400    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 12:14:27.707703 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701406    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 12:14:27.707703 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701412    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 12:14:27.707703 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701416    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 12:14:27.708043 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701420    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 12:14:27.708043 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701424    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 12:14:27.708043 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701428    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 12:14:27.708043 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701432    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 12:14:27.708043 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701436    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 12:14:27.708043 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701441    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 12:14:27.708043 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701445    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 12:14:27.708043 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701449    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 12:14:27.708043 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701453    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 12:14:27.708043 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701457    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 12:14:27.708043 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701461    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 12:14:27.708043 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701465    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 12:14:27.708043 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701470    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 12:14:27.708043 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701474    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 12:14:27.708043 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701478    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 12:14:27.708043 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701484    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 12:14:27.708043 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701488    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 12:14:27.708043 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701492    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 12:14:27.708043 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701496    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 12:14:27.708043 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701501    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 12:14:27.708043 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701505    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 12:14:27.708548 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701509    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 12:14:27.708548 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701513    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 12:14:27.708548 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701517    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 12:14:27.708548 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701521    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 12:14:27.708548 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701525    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 12:14:27.708548 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701530    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 12:14:27.708548 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701534    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 12:14:27.708548 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701567    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 12:14:27.708548 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701575    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 12:14:27.708548 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701580    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 12:14:27.708548 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701585    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 12:14:27.708548 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701590    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 12:14:27.708548 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701594    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 12:14:27.708548 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701601    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 12:14:27.708548 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701607    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 12:14:27.708548 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701612    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 12:14:27.708548 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701616    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 12:14:27.708548 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701620    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 12:14:27.708548 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701625    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 12:14:27.708548 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701629    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 12:14:27.709034 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701634    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 12:14:27.709034 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701638    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 12:14:27.709034 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701642    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 12:14:27.709034 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701646    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 12:14:27.709034 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701651    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 12:14:27.709034 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701655    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 12:14:27.709034 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701659    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 12:14:27.709034 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701664    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 12:14:27.709034 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701669    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 12:14:27.709034 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701673    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 12:14:27.709034 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701677    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 12:14:27.709034 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701681    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 12:14:27.709034 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701685    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 12:14:27.709034 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701688    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 12:14:27.709034 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701692    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 12:14:27.709034 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701696    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 12:14:27.709034 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701700    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 12:14:27.709034 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701704    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 12:14:27.709034 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701708    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 12:14:27.709579 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701713    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 12:14:27.709579 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701717    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 12:14:27.709579 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701721    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 12:14:27.709579 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701725    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 12:14:27.709579 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701729    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 12:14:27.709579 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701733    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 12:14:27.709579 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701737    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 12:14:27.709579 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701742    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 12:14:27.709579 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701748    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 12:14:27.709579 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701752    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 12:14:27.709579 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701758    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 12:14:27.709579 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701762    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 12:14:27.709579 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701766    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 12:14:27.709579 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701770    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 12:14:27.709579 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701774    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 12:14:27.709579 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701778    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 12:14:27.709579 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701782    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 12:14:27.709579 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701787    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 12:14:27.709579 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701791    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 12:14:27.709579 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701797    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 12:14:27.710060 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701802    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 12:14:27.710060 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701808    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 12:14:27.710060 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701812    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 12:14:27.710060 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701817    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 12:14:27.710060 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701821    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 12:14:27.710060 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:27.701825    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 12:14:27.710060 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.701833    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 12:14:27.710060 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.703963    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 12:14:27.710060 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.706012    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 12:14:27.710060 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.706999    2561 server.go:1019] "Starting client certificate rotation"
Sep 11 12:14:27.710060 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.707089    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 12:14:27.710060 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.707125    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 12:14:27.710060 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.709398    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 12:14:27.710482 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.710463    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 12:14:27.721054 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.721038    2561 log.go:25] "Validated CRI v1 runtime API"
Sep 11 12:14:27.723752 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.723739    2561 log.go:25] "Validated CRI v1 image API"
Sep 11 12:14:27.725139 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.725118    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 12:14:27.728662 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.728636    2561 fs.go:135] Filesystem UUIDs: map[5d848910-4575-4921-a08d-54e6092f51da:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c8c27fbb-b771-4a28-942a-dd6cde33cd53:/dev/nvme0n1p3]
Sep 11 12:14:27.728730 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.728661    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 12:14:27.733635 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.733526    2561 manager.go:216] Machine: {Timestamp:2026-09-11 12:14:27.732211807 +0000 UTC m=+0.325072914 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104450 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec217c90232fe88081b6b0bf66292fc3 SystemUUID:ec217c90-232f-e880-81b6-b0bf66292fc3 BootID:24f67bfd-44b7-49d8-814b-18d28132b1b5 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4f:0f:37:0e:b9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4f:0f:37:0e:b9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:75:1d:ca:bc:d7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 12:14:27.733635 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.733630    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 12:14:27.733744 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.733734    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 12:14:27.736150 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.736130    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 12:14:27.736276 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.736152    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-172.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 12:14:27.736913 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.736903    2561 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 12:14:27.736946 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.736916    2561 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 12:14:27.736946 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.736929    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 12:14:27.737521 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.737511    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 12:14:27.738275 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.738265    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 12:14:27.741732 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.741720    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 12:14:27.741857 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.741848    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 12:14:27.741893 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.741883    2561 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 12:14:27.741922 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.741907    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 12:14:27.741954 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.741929    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 12:14:27.741954 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.741941    2561 kubelet.go:398] "Adding apiserver pod source"
Sep 11 12:14:27.742026 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.742016    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 12:14:27.742474 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.742459    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 12:14:27.742474 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.742477    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 12:14:27.744641 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.744620    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 12:14:27.746963 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.746946    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 12:14:27.747058 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.746984    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 12:14:27.748663 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.748646    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 12:14:27.748728 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.748668    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 12:14:27.748728 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.748677    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 12:14:27.748728 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.748686    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 12:14:27.748728 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.748699    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 12:14:27.748728 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.748707    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 12:14:27.748728 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.748726    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 12:14:27.748906 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.748734    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 12:14:27.748906 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.748743    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 12:14:27.748906 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.748751    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 12:14:27.748906 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.748763    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 12:14:27.748906 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.748793    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 12:14:27.748906 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.748821    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 12:14:27.748906 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.748828    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 12:14:27.753677 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.753661    2561 server.go:1268] "Started kubelet"
Sep 11 12:14:27.753794 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.753759    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 12:14:27.753959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.753910    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 12:14:27.754044 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.753972    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 12:14:27.754409 ip-10-0-131-172 systemd[1]: Started Kubernetes Kubelet.
Sep 11 12:14:27.755745 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.755732    2561 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 12:14:27.756310 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.756290    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 12:14:27.756963 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.756947    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-172.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 12:14:27.757036 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:27.756984    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 12:14:27.757036 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:27.757022    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 12:14:27.759141 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:27.759120    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 12:14:27.760723 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.760709    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 12:14:27.760784 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.760727    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 12:14:27.760929 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:27.760851    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-172.ec2.internal\" not found"
Sep 11 12:14:27.761364 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.761340    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 12:14:27.761634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.761360    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 12:14:27.761634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.761491    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 12:14:27.761748 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.761699    2561 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 12:14:27.761748 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.761709    2561 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 12:14:27.761961 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.761904    2561 factory.go:55] Registering systemd factory
Sep 11 12:14:27.761961 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.761921    2561 factory.go:223] Registration of the systemd container factory successfully
Sep 11 12:14:27.762129 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.762116    2561 factory.go:153] Registering CRI-O factory
Sep 11 12:14:27.762183 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.762135    2561 factory.go:223] Registration of the crio container factory successfully
Sep 11 12:14:27.762233 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.762200    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 12:14:27.762233 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.762221    2561 factory.go:103] Registering Raw factory
Sep 11 12:14:27.762408 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.762236    2561 manager.go:1196] Started watching for new ooms in manager
Sep 11 12:14:27.762666 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.762651    2561 manager.go:349] Starting recovery of all containers
Sep 11 12:14:27.763275 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.763247    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 12:14:27.766439 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:27.766401    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 12:14:27.766881 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:27.766858    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 12:14:27.767499 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:27.766523    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-172.ec2.internal.18d44339dbda233f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-172.ec2.internal,UID:ip-10-0-131-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-172.ec2.internal,},FirstTimestamp:2026-09-11 12:14:27.753632575 +0000 UTC m=+0.346493682,LastTimestamp:2026-09-11 12:14:27.753632575 +0000 UTC m=+0.346493682,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-172.ec2.internal,}"
Sep 11 12:14:27.773183 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.773168    2561 manager.go:354] Recovery completed
Sep 11 12:14:27.780137 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.780121    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 12:14:27.782979 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.782935    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 12:14:27.783054 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.782986    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 12:14:27.783054 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.782999    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasSufficientPID"
Sep 11 12:14:27.783582 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.783566    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 12:14:27.783582 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.783584    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 12:14:27.783712 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.783590    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 12:14:27.783712 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.783601    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 12:14:27.784340 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.784323    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 12:14:27.784418 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.784350    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 12:14:27.784418 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.784362    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasSufficientPID"
Sep 11 12:14:27.784746 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.784732    2561 policy_none.go:49] "None policy: Start"
Sep 11 12:14:27.784811 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.784751    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 12:14:27.784811 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.784764    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 12:14:27.785760 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.785747    2561 policy_none.go:47] "Start"
Sep 11 12:14:27.787951 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:27.787885    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-172.ec2.internal.18d44339dd99c427  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-172.ec2.internal,UID:ip-10-0-131-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-172.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-172.ec2.internal,},FirstTimestamp:2026-09-11 12:14:27.782968359 +0000 UTC m=+0.375829474,LastTimestamp:2026-09-11 12:14:27.782968359 +0000 UTC m=+0.375829474,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-172.ec2.internal,}"
Sep 11 12:14:27.801008 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:27.800951    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-172.ec2.internal.18d44339dd9a25e8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-172.ec2.internal,UID:ip-10-0-131-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-172.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-172.ec2.internal,},FirstTimestamp:2026-09-11 12:14:27.782993384 +0000 UTC m=+0.375854490,LastTimestamp:2026-09-11 12:14:27.782993384 +0000 UTC m=+0.375854490,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-172.ec2.internal,}"
Sep 11 12:14:27.827143 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.811610    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tvzw7"
Sep 11 12:14:27.827143 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.812421    2561 manager.go:337] "Starting Device Plugin manager"
Sep 11 12:14:27.827143 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:27.812450    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 12:14:27.827143 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.812459    2561 server.go:87] "Starting device plugin registration server"
Sep 11 12:14:27.827143 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.812672    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 12:14:27.827143 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.812685    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 12:14:27.827143 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.812883    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 12:14:27.827143 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.813359    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 12:14:27.827143 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.813382    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 12:14:27.827143 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:27.813411    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 12:14:27.827143 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:27.813457    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-172.ec2.internal\" not found"
Sep 11 12:14:27.827143 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.820670    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tvzw7"
Sep 11 12:14:27.827143 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:27.823294    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-172.ec2.internal.18d44339dd9a4d73  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-172.ec2.internal,UID:ip-10-0-131-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-172.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-172.ec2.internal,},FirstTimestamp:2026-09-11 12:14:27.783003507 +0000 UTC m=+0.375864615,LastTimestamp:2026-09-11 12:14:27.783003507 +0000 UTC m=+0.375864615,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-172.ec2.internal,}"
Sep 11 12:14:27.844609 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:27.844550    2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-172.ec2.internal.18d44339dd99c427\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-172.ec2.internal.18d44339dd99c427  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-172.ec2.internal,UID:ip-10-0-131-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-172.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-172.ec2.internal,},FirstTimestamp:2026-09-11 12:14:27.782968359 +0000 UTC m=+0.375829474,LastTimestamp:2026-09-11 12:14:27.784338614 +0000 UTC m=+0.377199726,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-172.ec2.internal,}"
Sep 11 12:14:27.866301 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:27.866236    2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-172.ec2.internal.18d44339dd9a25e8\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-172.ec2.internal.18d44339dd9a25e8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-172.ec2.internal,UID:ip-10-0-131-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-172.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-172.ec2.internal,},FirstTimestamp:2026-09-11 12:14:27.782993384 +0000 UTC m=+0.375854490,LastTimestamp:2026-09-11 12:14:27.784355872 +0000 UTC m=+0.377216979,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-172.ec2.internal,}"
Sep 11 12:14:27.878857 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:27.878796    2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-172.ec2.internal.18d44339dd9a4d73\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-172.ec2.internal.18d44339dd9a4d73  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-172.ec2.internal,UID:ip-10-0-131-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-172.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-172.ec2.internal,},FirstTimestamp:2026-09-11 12:14:27.783003507 +0000 UTC m=+0.375864615,LastTimestamp:2026-09-11 12:14:27.784365419 +0000 UTC m=+0.377226526,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-172.ec2.internal,}"
Sep 11 12:14:27.888275 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:27.888223    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-172.ec2.internal.18d44339e013950a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-172.ec2.internal,UID:ip-10-0-131-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-131-172.ec2.internal,},FirstTimestamp:2026-09-11 12:14:27.824506122 +0000 UTC m=+0.417367216,LastTimestamp:2026-09-11 12:14:27.824506122 +0000 UTC m=+0.417367216,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-172.ec2.internal,}"
Sep 11 12:14:27.894192 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.894172    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 12:14:27.895509 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.895489    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 12:14:27.895579 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.895532    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 12:14:27.895579 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.895555    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 12:14:27.895768 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:27.895631    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 12:14:27.905698 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:27.905675    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 11 12:14:27.913724 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.913700    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 12:14:27.914433 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.914417    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 12:14:27.914507 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.914447    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 12:14:27.914507 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.914459    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasSufficientPID"
Sep 11 12:14:27.914507 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.914482    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-172.ec2.internal"
Sep 11 12:14:27.928180 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:27.928099    2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-172.ec2.internal.18d44339dd99c427\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-172.ec2.internal.18d44339dd99c427  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-172.ec2.internal,UID:ip-10-0-131-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-172.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-172.ec2.internal,},FirstTimestamp:2026-09-11 12:14:27.782968359 +0000 UTC m=+0.375829474,LastTimestamp:2026-09-11 12:14:27.914433218 +0000 UTC m=+0.507294332,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-172.ec2.internal,}"
Sep 11 12:14:27.938965 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:27.938922    2561 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-172.ec2.internal\" not found" node="ip-10-0-131-172.ec2.internal"
Sep 11 12:14:27.975911 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:27.975890    2561 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-172.ec2.internal\" not found" node="ip-10-0-131-172.ec2.internal"
Sep 11 12:14:27.996037 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.996007    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-172.ec2.internal"]
Sep 11 12:14:27.996088 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.996078    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 12:14:27.996871 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.996858    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 12:14:27.996920 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.996885    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 12:14:27.996920 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.996895    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasSufficientPID"
Sep 11 12:14:27.998253 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.998241    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 12:14:27.998370 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.998357    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.ec2.internal"
Sep 11 12:14:27.998416 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.998383    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 12:14:27.998893 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.998879    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 12:14:27.998946 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.998911    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 12:14:27.998946 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.998923    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasSufficientPID"
Sep 11 12:14:27.999019 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.998881    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 12:14:27.999019 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.998975    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 12:14:27.999019 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:27.998985    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasSufficientPID"
Sep 11 12:14:28.000201 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:28.000188    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.ec2.internal"
Sep 11 12:14:28.000252 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:28.000210    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 12:14:28.000936 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:28.000921    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 12:14:28.000992 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:28.000951    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 12:14:28.000992 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:28.000966    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasSufficientPID"
Sep 11 12:14:28.015637 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:28.015617    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-172.ec2.internal\" not found" node="ip-10-0-131-172.ec2.internal"
Sep 11 12:14:28.019428 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:28.019412    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-172.ec2.internal\" not found" node="ip-10-0-131-172.ec2.internal"
Sep 11 12:14:28.063342 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:28.063326    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/916ee7326c953598ae16ef7c8e25eae2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-172.ec2.internal\" (UID: \"916ee7326c953598ae16ef7c8e25eae2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.ec2.internal"
Sep 11 12:14:28.063425 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:28.063350    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/916ee7326c953598ae16ef7c8e25eae2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-172.ec2.internal\" (UID: \"916ee7326c953598ae16ef7c8e25eae2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.ec2.internal"
Sep 11 12:14:28.063425 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:28.063366    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4d587c23d1130ab850ab1128833ea53c-config\") pod \"kube-apiserver-proxy-ip-10-0-131-172.ec2.internal\" (UID: \"4d587c23d1130ab850ab1128833ea53c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.ec2.internal"
Sep 11 12:14:28.139320 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:28.139300    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 12:14:28.140073 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:28.140059    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 12:14:28.140152 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:28.140085    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 12:14:28.140152 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:28.140094    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasSufficientPID"
Sep 11 12:14:28.140152 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:28.140135    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-172.ec2.internal"
Sep 11 12:14:28.162950 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:28.162932    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-172.ec2.internal"
Sep 11 12:14:28.163005 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:28.162951    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-172.ec2.internal\": node \"ip-10-0-131-172.ec2.internal\" not found"
Sep 11 12:14:28.164287 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:28.164271    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/916ee7326c953598ae16ef7c8e25eae2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-172.ec2.internal\" (UID: \"916ee7326c953598ae16ef7c8e25eae2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.ec2.internal"
Sep 11 12:14:28.164329 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:28.164295    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/916ee7326c953598ae16ef7c8e25eae2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-172.ec2.internal\" (UID: \"916ee7326c953598ae16ef7c8e25eae2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.ec2.internal"
Sep 11 12:14:28.164329 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:28.164311    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4d587c23d1130ab850ab1128833ea53c-config\") pod \"kube-apiserver-proxy-ip-10-0-131-172.ec2.internal\" (UID: \"4d587c23d1130ab850ab1128833ea53c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.ec2.internal"
Sep 11 12:14:28.164401 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:28.164367    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4d587c23d1130ab850ab1128833ea53c-config\") pod \"kube-apiserver-proxy-ip-10-0-131-172.ec2.internal\" (UID: \"4d587c23d1130ab850ab1128833ea53c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.ec2.internal"
Sep 11 12:14:28.164401 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:28.164372    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/916ee7326c953598ae16ef7c8e25eae2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-172.ec2.internal\" (UID: \"916ee7326c953598ae16ef7c8e25eae2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.ec2.internal"
Sep 11 12:14:28.164473 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:28.164367    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/916ee7326c953598ae16ef7c8e25eae2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-172.ec2.internal\" (UID: \"916ee7326c953598ae16ef7c8e25eae2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.ec2.internal"
Sep 11 12:14:28.194704 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:28.194656    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-172.ec2.internal\" not found"
Sep 11 12:14:28.295524 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:28.295481    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-172.ec2.internal\" not found"
Sep 11 12:14:28.318623 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:28.318600    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.ec2.internal"
Sep 11 12:14:28.322689 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:28.322445    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.ec2.internal"
Sep 11 12:14:29.426605 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:29.426582    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-172.ec2.internal\" not found"
Sep 11 12:14:29.527079 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:29.527005    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-172.ec2.internal\" not found"
Sep 11 12:14:29.627514 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:29.627484    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-172.ec2.internal\" not found"
Sep 11 12:14:29.728007 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:29.727973    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-172.ec2.internal\" not found"
Sep 11 12:14:29.737141 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:29.737119    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 12:14:29.827831 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:29.827799    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 12:14:29.828878 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:29.828859    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-172.ec2.internal\" not found"
Sep 11 12:14:29.882074 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:29.882041    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d587c23d1130ab850ab1128833ea53c.slice/crio-f8e69b5cacfc1bb47b1b09be09d6fa152f76a541327ea6c2fc1ae4da6ad5a302 WatchSource:0}: Error finding container f8e69b5cacfc1bb47b1b09be09d6fa152f76a541327ea6c2fc1ae4da6ad5a302: Status 404 returned error can't find the container with id f8e69b5cacfc1bb47b1b09be09d6fa152f76a541327ea6c2fc1ae4da6ad5a302
Sep 11 12:14:29.886881 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:29.886866    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 12:14:29.928878 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:29.928841    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.ec2.internal" event={"ID":"4d587c23d1130ab850ab1128833ea53c","Type":"ContainerStarted","Data":"f8e69b5cacfc1bb47b1b09be09d6fa152f76a541327ea6c2fc1ae4da6ad5a302"}
Sep 11 12:14:29.928966 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:29.928910    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-172.ec2.internal\" not found"
Sep 11 12:14:29.933047 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:29.933035    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 12:14:29.933740 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:29.933728    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 12:14:29.933829 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:29.933758    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 12:14:29.933829 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:29.933776    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeHasSufficientPID"
Sep 11 12:14:29.960338 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:29.960313    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 12:09:28 +0000 UTC" deadline="2028-04-30 08:59:31.012264307 +0000 UTC"
Sep 11 12:14:29.960403 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:29.960339    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14324h45m1.051932255s"
Sep 11 12:14:29.987952 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:29.987934    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 12:14:30.029832 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:30.029810    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-172.ec2.internal\" not found"
Sep 11 12:14:30.045461 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:30.045435    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916ee7326c953598ae16ef7c8e25eae2.slice/crio-9a82151c2aa4d9c4334579fb8e0016d27ab1cfd718db6668668d6bede0e06125 WatchSource:0}: Error finding container 9a82151c2aa4d9c4334579fb8e0016d27ab1cfd718db6668668d6bede0e06125: Status 404 returned error can't find the container with id 9a82151c2aa4d9c4334579fb8e0016d27ab1cfd718db6668668d6bede0e06125
Sep 11 12:14:30.068503 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.068482    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 12:14:30.130088 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:30.130058    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-172.ec2.internal\" not found"
Sep 11 12:14:30.148696 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.148679    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 12:14:30.192235 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.192216    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.ec2.internal"
Sep 11 12:14:30.211133 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.211115    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 12:14:30.212218 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.212206    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.ec2.internal"
Sep 11 12:14:30.223447 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.223432    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 12:14:30.773075 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.773043    2561 apiserver.go:52] "Watching apiserver"
Sep 11 12:14:30.782223 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.782185    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 12:14:30.782545 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.782524    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-6qbjc","kube-system/konnectivity-agent-zbbqf","openshift-cluster-node-tuning-operator/tuned-trl89","openshift-multus/multus-k7m7m","openshift-network-diagnostics/network-check-target-kjxhz","openshift-network-operator/iptables-alerter-rzwzp","openshift-ovn-kubernetes/ovnkube-node-bcfm6","kube-system/kube-apiserver-proxy-ip-10-0-131-172.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p","openshift-dns/node-resolver-cj75q","openshift-image-registry/node-ca-5bxb8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.ec2.internal","openshift-multus/multus-additional-cni-plugins-vw24w","openshift-multus/network-metrics-daemon-bdr64"]
Sep 11 12:14:30.784177 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.784154    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.785525 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.785505    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vw24w"
Sep 11 12:14:30.786735 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.786715    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.787927 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.787911    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.789228 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.789208    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:14:30.789327 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:30.789287    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjxhz" podUID="719c0ee1-cbea-489a-9591-dcfb3e68b4e1"
Sep 11 12:14:30.789813 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.789791    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 12:14:30.789892 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.789831    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 12:14:30.790635 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.790616    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rzwzp"
Sep 11 12:14:30.791406 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.791392    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 12:14:30.792029 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.792013    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p"
Sep 11 12:14:30.793051 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.793032    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lvqds\""
Sep 11 12:14:30.793169 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.793145    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2r7jt\""
Sep 11 12:14:30.793339 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.793322    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 12:14:30.793642 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.793626    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:30.793716 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:30.793683    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qbjc" podUID="b254d6a2-6adc-468d-b21c-fb5ca9232417"
Sep 11 12:14:30.794324 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.794304    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 12:14:30.794477 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.794458    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 12:14:30.794572 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.794497    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 12:14:30.794572 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.794526    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rs9r5\""
Sep 11 12:14:30.794806 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.794776    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 12:14:30.794927 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.794913    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 12:14:30.795680 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.795662    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zbbqf"
Sep 11 12:14:30.796137 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.796120    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 12:14:30.796991 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.796971    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cj75q"
Sep 11 12:14:30.798291 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.798271    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5bxb8"
Sep 11 12:14:30.799078 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.799060    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 12:14:30.799161 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.799132    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 12:14:30.799542 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.799522    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:30.799631 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:30.799590    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdr64" podUID="26ca688b-efaa-42de-b332-78df0daf026b"
Sep 11 12:14:30.799853 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.799805    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 12:14:30.799933 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.799882    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n6hql\""
Sep 11 12:14:30.800132 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.800111    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 12:14:30.800260 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.800155    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 12:14:30.800387 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.800370    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 12:14:30.800471 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.800455    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 12:14:30.800566 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.800549    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 12:14:30.800748 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.800730    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 12:14:30.801294 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.801276    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 12:14:30.804541 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.802147    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 12:14:30.804541 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.802287    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-frznq\""
Sep 11 12:14:30.804541 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.803044    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h85m7\""
Sep 11 12:14:30.804541 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.803071    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 12:14:30.804541 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.803129    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-run-ovn\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.804541 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.803171    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5306d10e-12c5-4a0b-b048-c24edb6eed1a-iptables-alerter-script\") pod \"iptables-alerter-rzwzp\" (UID: \"5306d10e-12c5-4a0b-b048-c24edb6eed1a\") " pod="openshift-network-operator/iptables-alerter-rzwzp"
Sep 11 12:14:30.804541 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.803220    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-run-openvswitch\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.804541 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.803259    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vw24w\" (UID: \"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f\") " pod="openshift-multus/multus-additional-cni-plugins-vw24w"
Sep 11 12:14:30.804541 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.803294    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2r6g\" (UniqueName: \"kubernetes.io/projected/719c0ee1-cbea-489a-9591-dcfb3e68b4e1-kube-api-access-r2r6g\") pod \"network-check-target-kjxhz\" (UID: \"719c0ee1-cbea-489a-9591-dcfb3e68b4e1\") " pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:14:30.804541 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.803327    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-etc-sysconfig\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.804541 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.803484    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-lib-modules\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.804541 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.803510    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-multus-conf-dir\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.804541 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.803621    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7aa2dd34-9727-4afd-aae4-066d9cf11e43-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b7f5p\" (UID: \"7aa2dd34-9727-4afd-aae4-066d9cf11e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p"
Sep 11 12:14:30.804541 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.803655    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-log-socket\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.804541 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.803691    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vw24w\" (UID: \"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f\") " pod="openshift-multus/multus-additional-cni-plugins-vw24w"
Sep 11 12:14:30.804541 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.803763    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 12:14:30.804541 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.803800    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-etc-sysctl-d\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.804541 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.803825    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-sys\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.804541 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.803853    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-multus-cni-dir\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.805430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.803882    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7aa2dd34-9727-4afd-aae4-066d9cf11e43-sys-fs\") pod \"aws-ebs-csi-driver-node-b7f5p\" (UID: \"7aa2dd34-9727-4afd-aae4-066d9cf11e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p"
Sep 11 12:14:30.805430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.803997    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-var-lib-kubelet\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.805430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.804027    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/15eb8bc9-5975-4611-8543-1886caaf3a2f-multus-daemon-config\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.805430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.804050    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8nq9\" (UniqueName: \"kubernetes.io/projected/15eb8bc9-5975-4611-8543-1886caaf3a2f-kube-api-access-t8nq9\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.805430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.804124    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7aa2dd34-9727-4afd-aae4-066d9cf11e43-registration-dir\") pod \"aws-ebs-csi-driver-node-b7f5p\" (UID: \"7aa2dd34-9727-4afd-aae4-066d9cf11e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p"
Sep 11 12:14:30.805430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.804154    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7aa2dd34-9727-4afd-aae4-066d9cf11e43-device-dir\") pod \"aws-ebs-csi-driver-node-b7f5p\" (UID: \"7aa2dd34-9727-4afd-aae4-066d9cf11e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p"
Sep 11 12:14:30.805430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.804279    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-etc-modprobe-d\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.805430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.804309    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-run\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.805430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.804577    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-host-var-lib-kubelet\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.805430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.804648    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-host-kubelet\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.805430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.804730    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-var-lib-openvswitch\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.805430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.804771    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-hostroot\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.805430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.804805    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-etc-kubernetes\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.805430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.804850    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/711fdd9a-1bba-4eac-a618-f4863c741432-etc-tuned\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.805430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.804883    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlqgb\" (UniqueName: \"kubernetes.io/projected/711fdd9a-1bba-4eac-a618-f4863c741432-kube-api-access-wlqgb\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.805430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.804910    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7aa2dd34-9727-4afd-aae4-066d9cf11e43-socket-dir\") pod \"aws-ebs-csi-driver-node-b7f5p\" (UID: \"7aa2dd34-9727-4afd-aae4-066d9cf11e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p"
Sep 11 12:14:30.806099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.804963    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qtfp\" (UniqueName: \"kubernetes.io/projected/7aa2dd34-9727-4afd-aae4-066d9cf11e43-kube-api-access-5qtfp\") pod \"aws-ebs-csi-driver-node-b7f5p\" (UID: \"7aa2dd34-9727-4afd-aae4-066d9cf11e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p"
Sep 11 12:14:30.806099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.804990    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88402687-4e5b-411c-be7a-4cb955f90425-ovn-node-metrics-cert\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.806099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.805061    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-system-cni-dir\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.806099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.805096    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-multus-socket-dir-parent\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.806099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.805134    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-host-slash\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.806099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.805223    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f-system-cni-dir\") pod \"multus-additional-cni-plugins-vw24w\" (UID: \"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f\") " pod="openshift-multus/multus-additional-cni-plugins-vw24w"
Sep 11 12:14:30.806099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.805275    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-etc-kubernetes\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.806099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.805337    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7aa2dd34-9727-4afd-aae4-066d9cf11e43-etc-selinux\") pod \"aws-ebs-csi-driver-node-b7f5p\" (UID: \"7aa2dd34-9727-4afd-aae4-066d9cf11e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p"
Sep 11 12:14:30.806099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.805366    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-host-run-netns\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.806099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.805395    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qvp8\" (UniqueName: \"kubernetes.io/projected/88402687-4e5b-411c-be7a-4cb955f90425-kube-api-access-5qvp8\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.806099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.805422    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-host\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.806099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.805669    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15eb8bc9-5975-4611-8543-1886caaf3a2f-cni-binary-copy\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.806099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.805745    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-host-cni-bin\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.806099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.805791    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 12:14:30.806099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.805854    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f-cni-binary-copy\") pod \"multus-additional-cni-plugins-vw24w\" (UID: \"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f\") " pod="openshift-multus/multus-additional-cni-plugins-vw24w"
Sep 11 12:14:30.806099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.805884    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/711fdd9a-1bba-4eac-a618-f4863c741432-tmp\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.806099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.805985    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-host-run-multus-certs\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.806866 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.806019    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqggj\" (UniqueName: \"kubernetes.io/projected/5306d10e-12c5-4a0b-b048-c24edb6eed1a-kube-api-access-rqggj\") pod \"iptables-alerter-rzwzp\" (UID: \"5306d10e-12c5-4a0b-b048-c24edb6eed1a\") " pod="openshift-network-operator/iptables-alerter-rzwzp"
Sep 11 12:14:30.806866 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.806071    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 12:14:30.806866 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.806154    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-host-var-lib-cni-bin\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.806866 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.806321    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w9vbg\""
Sep 11 12:14:30.806866 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.806518    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 12:14:30.806866 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.806531    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 12:14:30.806866 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.806804    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pd8gb\""
Sep 11 12:14:30.806866 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.806861    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 12:14:30.807325 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.806243    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-host-var-lib-cni-multus\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.807325 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.807001    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-systemd-units\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.807325 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.807034    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-etc-openvswitch\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.807325 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.807077    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88402687-4e5b-411c-be7a-4cb955f90425-ovnkube-config\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.807325 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.807095    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88402687-4e5b-411c-be7a-4cb955f90425-ovnkube-script-lib\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.807325 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.807119    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75s2n\" (UniqueName: \"kubernetes.io/projected/0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f-kube-api-access-75s2n\") pod \"multus-additional-cni-plugins-vw24w\" (UID: \"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f\") " pod="openshift-multus/multus-additional-cni-plugins-vw24w"
Sep 11 12:14:30.807325 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.807182    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-run-systemd\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.808511 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.808146    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-node-log\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.808511 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.808214    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-host-cni-netd\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.808511 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.808243    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.808511 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.808269    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88402687-4e5b-411c-be7a-4cb955f90425-env-overrides\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.808511 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.808302    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lrgl6\""
Sep 11 12:14:30.808511 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.808314    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-etc-systemd\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.808511 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.808337    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-etc-sysctl-conf\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.808511 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.808377    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-host-run-netns\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.808511 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.808401    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-host-run-ovn-kubernetes\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.808511 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.808424    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f-cnibin\") pod \"multus-additional-cni-plugins-vw24w\" (UID: \"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f\") " pod="openshift-multus/multus-additional-cni-plugins-vw24w"
Sep 11 12:14:30.808511 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.808447    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vw24w\" (UID: \"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f\") " pod="openshift-multus/multus-additional-cni-plugins-vw24w"
Sep 11 12:14:30.808511 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.808488    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5306d10e-12c5-4a0b-b048-c24edb6eed1a-host-slash\") pod \"iptables-alerter-rzwzp\" (UID: \"5306d10e-12c5-4a0b-b048-c24edb6eed1a\") " pod="openshift-network-operator/iptables-alerter-rzwzp"
Sep 11 12:14:30.809076 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.808524    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-cnibin\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.809076 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.808552    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-host-run-k8s-cni-cncf-io\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.819394 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.819377    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 12:14:30.841512 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.841492    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xjnv9"
Sep 11 12:14:30.861267 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.861245    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xjnv9"
Sep 11 12:14:30.893476 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.893457    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 12:14:30.909336 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909318    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-systemd-units\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.909443 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909340    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-etc-openvswitch\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.909443 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909359    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88402687-4e5b-411c-be7a-4cb955f90425-ovnkube-config\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.909443 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909382    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88402687-4e5b-411c-be7a-4cb955f90425-ovnkube-script-lib\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.909443 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909402    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75s2n\" (UniqueName: \"kubernetes.io/projected/0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f-kube-api-access-75s2n\") pod \"multus-additional-cni-plugins-vw24w\" (UID: \"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f\") " pod="openshift-multus/multus-additional-cni-plugins-vw24w"
Sep 11 12:14:30.909443 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909416    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-run-systemd\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.909443 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909431    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-node-log\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.909443 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909445    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-host-cni-netd\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.909783 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909460    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.909783 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909475    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88402687-4e5b-411c-be7a-4cb955f90425-env-overrides\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.909783 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909505    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-etc-systemd\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.909783 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909529    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-etc-sysctl-conf\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.909783 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909555    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b5df1e85-4605-4418-a2cb-b88a26b5c5b3-hosts-file\") pod \"node-resolver-cj75q\" (UID: \"b5df1e85-4605-4418-a2cb-b88a26b5c5b3\") " pod="openshift-dns/node-resolver-cj75q"
Sep 11 12:14:30.909783 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909575    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-host-run-netns\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.909783 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909590    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-host-run-ovn-kubernetes\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.909783 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909604    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f-cnibin\") pod \"multus-additional-cni-plugins-vw24w\" (UID: \"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f\") " pod="openshift-multus/multus-additional-cni-plugins-vw24w"
Sep 11 12:14:30.909783 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909620    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vw24w\" (UID: \"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f\") " pod="openshift-multus/multus-additional-cni-plugins-vw24w"
Sep 11 12:14:30.909783 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909634    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/455e42df-85a6-4c25-a449-e12375d132e6-host\") pod \"node-ca-5bxb8\" (UID: \"455e42df-85a6-4c25-a449-e12375d132e6\") " pod="openshift-image-registry/node-ca-5bxb8"
Sep 11 12:14:30.909783 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909648    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5306d10e-12c5-4a0b-b048-c24edb6eed1a-host-slash\") pod \"iptables-alerter-rzwzp\" (UID: \"5306d10e-12c5-4a0b-b048-c24edb6eed1a\") " pod="openshift-network-operator/iptables-alerter-rzwzp"
Sep 11 12:14:30.909783 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909662    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-cnibin\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.909783 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909677    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-host-run-k8s-cni-cncf-io\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.909783 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909705    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-run-ovn\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.909783 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909742    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5306d10e-12c5-4a0b-b048-c24edb6eed1a-iptables-alerter-script\") pod \"iptables-alerter-rzwzp\" (UID: \"5306d10e-12c5-4a0b-b048-c24edb6eed1a\") " pod="openshift-network-operator/iptables-alerter-rzwzp"
Sep 11 12:14:30.909783 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909765    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-run-openvswitch\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.909783 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909788    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vw24w\" (UID: \"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f\") " pod="openshift-multus/multus-additional-cni-plugins-vw24w"
Sep 11 12:14:30.910545 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909806    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2r6g\" (UniqueName: \"kubernetes.io/projected/719c0ee1-cbea-489a-9591-dcfb3e68b4e1-kube-api-access-r2r6g\") pod \"network-check-target-kjxhz\" (UID: \"719c0ee1-cbea-489a-9591-dcfb3e68b4e1\") " pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:14:30.910545 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909835    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-etc-sysconfig\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.910545 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909852    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-lib-modules\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.910545 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909873    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-multus-conf-dir\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.910545 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909937    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7aa2dd34-9727-4afd-aae4-066d9cf11e43-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b7f5p\" (UID: \"7aa2dd34-9727-4afd-aae4-066d9cf11e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p"
Sep 11 12:14:30.910545 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909987    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-run-systemd\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.910545 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.909996    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f-cnibin\") pod \"multus-additional-cni-plugins-vw24w\" (UID: \"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f\") " pod="openshift-multus/multus-additional-cni-plugins-vw24w"
Sep 11 12:14:30.910545 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910000    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-cnibin\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.910545 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910000    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7aa2dd34-9727-4afd-aae4-066d9cf11e43-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b7f5p\" (UID: \"7aa2dd34-9727-4afd-aae4-066d9cf11e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p"
Sep 11 12:14:30.910545 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910024    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-host-run-netns\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.910545 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910033    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-host-run-ovn-kubernetes\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.910545 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910034    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-lib-modules\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.910545 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910054    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-node-log\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.910545 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910059    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-run-ovn\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.910545 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910061    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.910545 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910029    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-host-run-k8s-cni-cncf-io\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.910545 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910082    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-etc-sysconfig\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.910545 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910092    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-multus-conf-dir\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.911347 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910099    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-systemd-units\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.911347 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910100    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-etc-openvswitch\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.911347 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910117    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-host-cni-netd\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.911347 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910204    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-etc-sysctl-conf\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.911347 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910222    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b254d6a2-6adc-468d-b21c-fb5ca9232417-kubelet-config\") pod \"global-pull-secret-syncer-6qbjc\" (UID: \"b254d6a2-6adc-468d-b21c-fb5ca9232417\") " pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:30.911347 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910227    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-run-openvswitch\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.911347 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910254    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-log-socket\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.911347 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910353    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5306d10e-12c5-4a0b-b048-c24edb6eed1a-host-slash\") pod \"iptables-alerter-rzwzp\" (UID: \"5306d10e-12c5-4a0b-b048-c24edb6eed1a\") " pod="openshift-network-operator/iptables-alerter-rzwzp"
Sep 11 12:14:30.911347 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910400    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vw24w\" (UID: \"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f\") " pod="openshift-multus/multus-additional-cni-plugins-vw24w"
Sep 11 12:14:30.911347 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910428    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-etc-sysctl-d\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.911347 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910451    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-sys\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.911347 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910476    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-multus-cni-dir\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.911347 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910533    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7aa2dd34-9727-4afd-aae4-066d9cf11e43-sys-fs\") pod \"aws-ebs-csi-driver-node-b7f5p\" (UID: \"7aa2dd34-9727-4afd-aae4-066d9cf11e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p"
Sep 11 12:14:30.911347 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910577    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b254d6a2-6adc-468d-b21c-fb5ca9232417-dbus\") pod \"global-pull-secret-syncer-6qbjc\" (UID: \"b254d6a2-6adc-468d-b21c-fb5ca9232417\") " pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:30.911347 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910598    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88402687-4e5b-411c-be7a-4cb955f90425-env-overrides\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.911347 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910647    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-sys\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.911347 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910691    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7aa2dd34-9727-4afd-aae4-066d9cf11e43-sys-fs\") pod \"aws-ebs-csi-driver-node-b7f5p\" (UID: \"7aa2dd34-9727-4afd-aae4-066d9cf11e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p"
Sep 11 12:14:30.912029 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910702    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88402687-4e5b-411c-be7a-4cb955f90425-ovnkube-script-lib\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.912029 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910727    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-etc-sysctl-d\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.912029 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910602    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9rvs\" (UniqueName: \"kubernetes.io/projected/455e42df-85a6-4c25-a449-e12375d132e6-kube-api-access-w9rvs\") pod \"node-ca-5bxb8\" (UID: \"455e42df-85a6-4c25-a449-e12375d132e6\") " pod="openshift-image-registry/node-ca-5bxb8"
Sep 11 12:14:30.912029 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910744    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-log-socket\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.912029 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910743    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vw24w\" (UID: \"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f\") " pod="openshift-multus/multus-additional-cni-plugins-vw24w"
Sep 11 12:14:30.912029 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910750    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vw24w\" (UID: \"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f\") " pod="openshift-multus/multus-additional-cni-plugins-vw24w"
Sep 11 12:14:30.912029 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910767    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-var-lib-kubelet\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.912029 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910813    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-var-lib-kubelet\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.912029 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910817    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5306d10e-12c5-4a0b-b048-c24edb6eed1a-iptables-alerter-script\") pod \"iptables-alerter-rzwzp\" (UID: \"5306d10e-12c5-4a0b-b048-c24edb6eed1a\") " pod="openshift-network-operator/iptables-alerter-rzwzp"
Sep 11 12:14:30.912029 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910824    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/15eb8bc9-5975-4611-8543-1886caaf3a2f-multus-daemon-config\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.912029 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910854    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8nq9\" (UniqueName: \"kubernetes.io/projected/15eb8bc9-5975-4611-8543-1886caaf3a2f-kube-api-access-t8nq9\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.912029 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910881    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7aa2dd34-9727-4afd-aae4-066d9cf11e43-registration-dir\") pod \"aws-ebs-csi-driver-node-b7f5p\" (UID: \"7aa2dd34-9727-4afd-aae4-066d9cf11e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p"
Sep 11 12:14:30.912029 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910180    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-etc-systemd\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.912029 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910800    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-multus-cni-dir\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.912029 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910906    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7aa2dd34-9727-4afd-aae4-066d9cf11e43-device-dir\") pod \"aws-ebs-csi-driver-node-b7f5p\" (UID: \"7aa2dd34-9727-4afd-aae4-066d9cf11e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p"
Sep 11 12:14:30.912029 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910928    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88402687-4e5b-411c-be7a-4cb955f90425-ovnkube-config\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.912029 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910947    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b254d6a2-6adc-468d-b21c-fb5ca9232417-original-pull-secret\") pod \"global-pull-secret-syncer-6qbjc\" (UID: \"b254d6a2-6adc-468d-b21c-fb5ca9232417\") " pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:30.912814 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910956    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7aa2dd34-9727-4afd-aae4-066d9cf11e43-device-dir\") pod \"aws-ebs-csi-driver-node-b7f5p\" (UID: \"7aa2dd34-9727-4afd-aae4-066d9cf11e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p"
Sep 11 12:14:30.912814 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.910974    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/455e42df-85a6-4c25-a449-e12375d132e6-serviceca\") pod \"node-ca-5bxb8\" (UID: \"455e42df-85a6-4c25-a449-e12375d132e6\") " pod="openshift-image-registry/node-ca-5bxb8"
Sep 11 12:14:30.912814 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911001    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k8mj\" (UniqueName: \"kubernetes.io/projected/b5df1e85-4605-4418-a2cb-b88a26b5c5b3-kube-api-access-5k8mj\") pod \"node-resolver-cj75q\" (UID: \"b5df1e85-4605-4418-a2cb-b88a26b5c5b3\") " pod="openshift-dns/node-resolver-cj75q"
Sep 11 12:14:30.912814 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911028    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-etc-modprobe-d\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.912814 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911051    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-run\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.912814 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911061    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7aa2dd34-9727-4afd-aae4-066d9cf11e43-registration-dir\") pod \"aws-ebs-csi-driver-node-b7f5p\" (UID: \"7aa2dd34-9727-4afd-aae4-066d9cf11e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p"
Sep 11 12:14:30.912814 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911076    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-host-var-lib-kubelet\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.912814 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911099    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/28c2ca1a-9bbb-4cb5-a226-69eb4db972a2-agent-certs\") pod \"konnectivity-agent-zbbqf\" (UID: \"28c2ca1a-9bbb-4cb5-a226-69eb4db972a2\") " pod="kube-system/konnectivity-agent-zbbqf"
Sep 11 12:14:30.912814 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911110    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-run\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.912814 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911115    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-host-var-lib-kubelet\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.912814 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911124    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/28c2ca1a-9bbb-4cb5-a226-69eb4db972a2-konnectivity-ca\") pod \"konnectivity-agent-zbbqf\" (UID: \"28c2ca1a-9bbb-4cb5-a226-69eb4db972a2\") " pod="kube-system/konnectivity-agent-zbbqf"
Sep 11 12:14:30.912814 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911144    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-etc-modprobe-d\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.912814 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911152    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vw24w\" (UID: \"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f\") " pod="openshift-multus/multus-additional-cni-plugins-vw24w"
Sep 11 12:14:30.912814 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911176    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-host-kubelet\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.912814 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911220    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-var-lib-openvswitch\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.912814 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911237    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-host-kubelet\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.912814 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911244    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-hostroot\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.913758 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911268    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-etc-kubernetes\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.913758 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911271    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-var-lib-openvswitch\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.913758 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911295    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5df1e85-4605-4418-a2cb-b88a26b5c5b3-tmp-dir\") pod \"node-resolver-cj75q\" (UID: \"b5df1e85-4605-4418-a2cb-b88a26b5c5b3\") " pod="openshift-dns/node-resolver-cj75q"
Sep 11 12:14:30.913758 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911306    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/15eb8bc9-5975-4611-8543-1886caaf3a2f-multus-daemon-config\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.913758 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911319    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/711fdd9a-1bba-4eac-a618-f4863c741432-etc-tuned\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.913758 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911321    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-etc-kubernetes\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.913758 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911298    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-hostroot\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.913758 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911344    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlqgb\" (UniqueName: \"kubernetes.io/projected/711fdd9a-1bba-4eac-a618-f4863c741432-kube-api-access-wlqgb\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.913758 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911433    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7aa2dd34-9727-4afd-aae4-066d9cf11e43-socket-dir\") pod \"aws-ebs-csi-driver-node-b7f5p\" (UID: \"7aa2dd34-9727-4afd-aae4-066d9cf11e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p"
Sep 11 12:14:30.913758 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911600    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7aa2dd34-9727-4afd-aae4-066d9cf11e43-socket-dir\") pod \"aws-ebs-csi-driver-node-b7f5p\" (UID: \"7aa2dd34-9727-4afd-aae4-066d9cf11e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p"
Sep 11 12:14:30.913758 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911621    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qtfp\" (UniqueName: \"kubernetes.io/projected/7aa2dd34-9727-4afd-aae4-066d9cf11e43-kube-api-access-5qtfp\") pod \"aws-ebs-csi-driver-node-b7f5p\" (UID: \"7aa2dd34-9727-4afd-aae4-066d9cf11e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p"
Sep 11 12:14:30.913758 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911637    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 12:14:30.913758 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911658    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ca688b-efaa-42de-b332-78df0daf026b-metrics-certs\") pod \"network-metrics-daemon-bdr64\" (UID: \"26ca688b-efaa-42de-b332-78df0daf026b\") " pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:30.913758 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911704    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88402687-4e5b-411c-be7a-4cb955f90425-ovn-node-metrics-cert\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.913758 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911721    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-system-cni-dir\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.913758 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911735    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-multus-socket-dir-parent\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.913758 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911757    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltgnt\" (UniqueName: \"kubernetes.io/projected/26ca688b-efaa-42de-b332-78df0daf026b-kube-api-access-ltgnt\") pod \"network-metrics-daemon-bdr64\" (UID: \"26ca688b-efaa-42de-b332-78df0daf026b\") " pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:30.913758 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911782    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-host-slash\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.914634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911804    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f-system-cni-dir\") pod \"multus-additional-cni-plugins-vw24w\" (UID: \"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f\") " pod="openshift-multus/multus-additional-cni-plugins-vw24w"
Sep 11 12:14:30.914634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911827    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-etc-kubernetes\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.914634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911846    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7aa2dd34-9727-4afd-aae4-066d9cf11e43-etc-selinux\") pod \"aws-ebs-csi-driver-node-b7f5p\" (UID: \"7aa2dd34-9727-4afd-aae4-066d9cf11e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p"
Sep 11 12:14:30.914634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911870    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-host-run-netns\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.914634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911894    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qvp8\" (UniqueName: \"kubernetes.io/projected/88402687-4e5b-411c-be7a-4cb955f90425-kube-api-access-5qvp8\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.914634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911915    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-host\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.914634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911933    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-system-cni-dir\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.914634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911936    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15eb8bc9-5975-4611-8543-1886caaf3a2f-cni-binary-copy\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.914634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.911984    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-host-cni-bin\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.914634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.912013    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f-cni-binary-copy\") pod \"multus-additional-cni-plugins-vw24w\" (UID: \"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f\") " pod="openshift-multus/multus-additional-cni-plugins-vw24w"
Sep 11 12:14:30.914634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.912036    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/711fdd9a-1bba-4eac-a618-f4863c741432-tmp\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.914634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.912044    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7aa2dd34-9727-4afd-aae4-066d9cf11e43-etc-selinux\") pod \"aws-ebs-csi-driver-node-b7f5p\" (UID: \"7aa2dd34-9727-4afd-aae4-066d9cf11e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p"
Sep 11 12:14:30.914634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.912059    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-host-run-multus-certs\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.914634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.912085    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqggj\" (UniqueName: \"kubernetes.io/projected/5306d10e-12c5-4a0b-b048-c24edb6eed1a-kube-api-access-rqggj\") pod \"iptables-alerter-rzwzp\" (UID: \"5306d10e-12c5-4a0b-b048-c24edb6eed1a\") " pod="openshift-network-operator/iptables-alerter-rzwzp"
Sep 11 12:14:30.914634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.912094    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-host-slash\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.914634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.912109    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-host-var-lib-cni-bin\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.914634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.912151    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-host-var-lib-cni-bin\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.915287 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.912152    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-host-var-lib-cni-multus\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.915287 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.912206    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-host-var-lib-cni-multus\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.915287 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.912265    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-etc-kubernetes\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.915287 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.912264    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-host-run-netns\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.915287 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.912303    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88402687-4e5b-411c-be7a-4cb955f90425-host-cni-bin\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.915287 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.912343    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f-system-cni-dir\") pod \"multus-additional-cni-plugins-vw24w\" (UID: \"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f\") " pod="openshift-multus/multus-additional-cni-plugins-vw24w"
Sep 11 12:14:30.915287 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.912347    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15eb8bc9-5975-4611-8543-1886caaf3a2f-cni-binary-copy\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.915287 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.912411    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/711fdd9a-1bba-4eac-a618-f4863c741432-host\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.915287 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.912451    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-host-run-multus-certs\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.915287 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.912545    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/15eb8bc9-5975-4611-8543-1886caaf3a2f-multus-socket-dir-parent\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.915287 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.912786    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f-cni-binary-copy\") pod \"multus-additional-cni-plugins-vw24w\" (UID: \"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f\") " pod="openshift-multus/multus-additional-cni-plugins-vw24w"
Sep 11 12:14:30.915287 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.914996    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/711fdd9a-1bba-4eac-a618-f4863c741432-etc-tuned\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.915287 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.915065    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/711fdd9a-1bba-4eac-a618-f4863c741432-tmp\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.915287 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.915159    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88402687-4e5b-411c-be7a-4cb955f90425-ovn-node-metrics-cert\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.931041 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.931017    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.ec2.internal" event={"ID":"916ee7326c953598ae16ef7c8e25eae2","Type":"ContainerStarted","Data":"9a82151c2aa4d9c4334579fb8e0016d27ab1cfd718db6668668d6bede0e06125"}
Sep 11 12:14:30.937647 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.937624    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qtfp\" (UniqueName: \"kubernetes.io/projected/7aa2dd34-9727-4afd-aae4-066d9cf11e43-kube-api-access-5qtfp\") pod \"aws-ebs-csi-driver-node-b7f5p\" (UID: \"7aa2dd34-9727-4afd-aae4-066d9cf11e43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p"
Sep 11 12:14:30.938014 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.937982    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8nq9\" (UniqueName: \"kubernetes.io/projected/15eb8bc9-5975-4611-8543-1886caaf3a2f-kube-api-access-t8nq9\") pod \"multus-k7m7m\" (UID: \"15eb8bc9-5975-4611-8543-1886caaf3a2f\") " pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:30.938806 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:30.938783    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 12:14:30.938806 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:30.938810    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 12:14:30.938966 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:30.938823    2561 projected.go:196] Error preparing data for projected volume kube-api-access-r2r6g for pod openshift-network-diagnostics/network-check-target-kjxhz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:30.938966 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:30.938897    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/719c0ee1-cbea-489a-9591-dcfb3e68b4e1-kube-api-access-r2r6g podName:719c0ee1-cbea-489a-9591-dcfb3e68b4e1 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:31.438878584 +0000 UTC m=+3.001658339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r2r6g" (UniqueName: "kubernetes.io/projected/719c0ee1-cbea-489a-9591-dcfb3e68b4e1-kube-api-access-r2r6g") pod "network-check-target-kjxhz" (UID: "719c0ee1-cbea-489a-9591-dcfb3e68b4e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:30.939304 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.939277    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlqgb\" (UniqueName: \"kubernetes.io/projected/711fdd9a-1bba-4eac-a618-f4863c741432-kube-api-access-wlqgb\") pod \"tuned-trl89\" (UID: \"711fdd9a-1bba-4eac-a618-f4863c741432\") " pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:30.939409 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.939393    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75s2n\" (UniqueName: \"kubernetes.io/projected/0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f-kube-api-access-75s2n\") pod \"multus-additional-cni-plugins-vw24w\" (UID: \"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f\") " pod="openshift-multus/multus-additional-cni-plugins-vw24w"
Sep 11 12:14:30.939647 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.939629    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qvp8\" (UniqueName: \"kubernetes.io/projected/88402687-4e5b-411c-be7a-4cb955f90425-kube-api-access-5qvp8\") pod \"ovnkube-node-bcfm6\" (UID: \"88402687-4e5b-411c-be7a-4cb955f90425\") " pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:30.940231 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:30.940213    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqggj\" (UniqueName: \"kubernetes.io/projected/5306d10e-12c5-4a0b-b048-c24edb6eed1a-kube-api-access-rqggj\") pod \"iptables-alerter-rzwzp\" (UID: \"5306d10e-12c5-4a0b-b048-c24edb6eed1a\") " pod="openshift-network-operator/iptables-alerter-rzwzp"
Sep 11 12:14:31.012915 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.012889    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b254d6a2-6adc-468d-b21c-fb5ca9232417-original-pull-secret\") pod \"global-pull-secret-syncer-6qbjc\" (UID: \"b254d6a2-6adc-468d-b21c-fb5ca9232417\") " pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:31.013056 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.012924    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/455e42df-85a6-4c25-a449-e12375d132e6-serviceca\") pod \"node-ca-5bxb8\" (UID: \"455e42df-85a6-4c25-a449-e12375d132e6\") " pod="openshift-image-registry/node-ca-5bxb8"
Sep 11 12:14:31.013056 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.012941    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k8mj\" (UniqueName: \"kubernetes.io/projected/b5df1e85-4605-4418-a2cb-b88a26b5c5b3-kube-api-access-5k8mj\") pod \"node-resolver-cj75q\" (UID: \"b5df1e85-4605-4418-a2cb-b88a26b5c5b3\") " pod="openshift-dns/node-resolver-cj75q"
Sep 11 12:14:31.013056 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.012957    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/28c2ca1a-9bbb-4cb5-a226-69eb4db972a2-agent-certs\") pod \"konnectivity-agent-zbbqf\" (UID: \"28c2ca1a-9bbb-4cb5-a226-69eb4db972a2\") " pod="kube-system/konnectivity-agent-zbbqf"
Sep 11 12:14:31.013056 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.012971    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/28c2ca1a-9bbb-4cb5-a226-69eb4db972a2-konnectivity-ca\") pod \"konnectivity-agent-zbbqf\" (UID: \"28c2ca1a-9bbb-4cb5-a226-69eb4db972a2\") " pod="kube-system/konnectivity-agent-zbbqf"
Sep 11 12:14:31.013056 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.012987    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5df1e85-4605-4418-a2cb-b88a26b5c5b3-tmp-dir\") pod \"node-resolver-cj75q\" (UID: \"b5df1e85-4605-4418-a2cb-b88a26b5c5b3\") " pod="openshift-dns/node-resolver-cj75q"
Sep 11 12:14:31.013056 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.013012    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ca688b-efaa-42de-b332-78df0daf026b-metrics-certs\") pod \"network-metrics-daemon-bdr64\" (UID: \"26ca688b-efaa-42de-b332-78df0daf026b\") " pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:31.013418 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:31.013083    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:31.013418 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:31.013129    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ca688b-efaa-42de-b332-78df0daf026b-metrics-certs podName:26ca688b-efaa-42de-b332-78df0daf026b nodeName:}" failed. No retries permitted until 2026-09-11 12:14:31.513115423 +0000 UTC m=+3.075895163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26ca688b-efaa-42de-b332-78df0daf026b-metrics-certs") pod "network-metrics-daemon-bdr64" (UID: "26ca688b-efaa-42de-b332-78df0daf026b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:31.013418 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:31.013325    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:31.013418 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:31.013362    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b254d6a2-6adc-468d-b21c-fb5ca9232417-original-pull-secret podName:b254d6a2-6adc-468d-b21c-fb5ca9232417 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:31.513350739 +0000 UTC m=+3.076130493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b254d6a2-6adc-468d-b21c-fb5ca9232417-original-pull-secret") pod "global-pull-secret-syncer-6qbjc" (UID: "b254d6a2-6adc-468d-b21c-fb5ca9232417") : object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:31.013591 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.013422    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/455e42df-85a6-4c25-a449-e12375d132e6-serviceca\") pod \"node-ca-5bxb8\" (UID: \"455e42df-85a6-4c25-a449-e12375d132e6\") " pod="openshift-image-registry/node-ca-5bxb8"
Sep 11 12:14:31.013591 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.013438    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltgnt\" (UniqueName: \"kubernetes.io/projected/26ca688b-efaa-42de-b332-78df0daf026b-kube-api-access-ltgnt\") pod \"network-metrics-daemon-bdr64\" (UID: \"26ca688b-efaa-42de-b332-78df0daf026b\") " pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:31.013591 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.013478    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5df1e85-4605-4418-a2cb-b88a26b5c5b3-tmp-dir\") pod \"node-resolver-cj75q\" (UID: \"b5df1e85-4605-4418-a2cb-b88a26b5c5b3\") " pod="openshift-dns/node-resolver-cj75q"
Sep 11 12:14:31.013591 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.013496    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b5df1e85-4605-4418-a2cb-b88a26b5c5b3-hosts-file\") pod \"node-resolver-cj75q\" (UID: \"b5df1e85-4605-4418-a2cb-b88a26b5c5b3\") " pod="openshift-dns/node-resolver-cj75q"
Sep 11 12:14:31.013591 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.013526    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/455e42df-85a6-4c25-a449-e12375d132e6-host\") pod \"node-ca-5bxb8\" (UID: \"455e42df-85a6-4c25-a449-e12375d132e6\") " pod="openshift-image-registry/node-ca-5bxb8"
Sep 11 12:14:31.013591 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.013566    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/28c2ca1a-9bbb-4cb5-a226-69eb4db972a2-konnectivity-ca\") pod \"konnectivity-agent-zbbqf\" (UID: \"28c2ca1a-9bbb-4cb5-a226-69eb4db972a2\") " pod="kube-system/konnectivity-agent-zbbqf"
Sep 11 12:14:31.013591 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.013583    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b5df1e85-4605-4418-a2cb-b88a26b5c5b3-hosts-file\") pod \"node-resolver-cj75q\" (UID: \"b5df1e85-4605-4418-a2cb-b88a26b5c5b3\") " pod="openshift-dns/node-resolver-cj75q"
Sep 11 12:14:31.013857 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.013572    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b254d6a2-6adc-468d-b21c-fb5ca9232417-kubelet-config\") pod \"global-pull-secret-syncer-6qbjc\" (UID: \"b254d6a2-6adc-468d-b21c-fb5ca9232417\") " pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:31.013857 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.013608    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/455e42df-85a6-4c25-a449-e12375d132e6-host\") pod \"node-ca-5bxb8\" (UID: \"455e42df-85a6-4c25-a449-e12375d132e6\") " pod="openshift-image-registry/node-ca-5bxb8"
Sep 11 12:14:31.013857 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.013616    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b254d6a2-6adc-468d-b21c-fb5ca9232417-kubelet-config\") pod \"global-pull-secret-syncer-6qbjc\" (UID: \"b254d6a2-6adc-468d-b21c-fb5ca9232417\") " pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:31.013857 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.013634    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b254d6a2-6adc-468d-b21c-fb5ca9232417-dbus\") pod \"global-pull-secret-syncer-6qbjc\" (UID: \"b254d6a2-6adc-468d-b21c-fb5ca9232417\") " pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:31.013857 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.013662    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rvs\" (UniqueName: \"kubernetes.io/projected/455e42df-85a6-4c25-a449-e12375d132e6-kube-api-access-w9rvs\") pod \"node-ca-5bxb8\" (UID: \"455e42df-85a6-4c25-a449-e12375d132e6\") " pod="openshift-image-registry/node-ca-5bxb8"
Sep 11 12:14:31.013857 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.013732    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b254d6a2-6adc-468d-b21c-fb5ca9232417-dbus\") pod \"global-pull-secret-syncer-6qbjc\" (UID: \"b254d6a2-6adc-468d-b21c-fb5ca9232417\") " pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:31.015345 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.015322    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/28c2ca1a-9bbb-4cb5-a226-69eb4db972a2-agent-certs\") pod \"konnectivity-agent-zbbqf\" (UID: \"28c2ca1a-9bbb-4cb5-a226-69eb4db972a2\") " pod="kube-system/konnectivity-agent-zbbqf"
Sep 11 12:14:31.024538 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.024490    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k8mj\" (UniqueName: \"kubernetes.io/projected/b5df1e85-4605-4418-a2cb-b88a26b5c5b3-kube-api-access-5k8mj\") pod \"node-resolver-cj75q\" (UID: \"b5df1e85-4605-4418-a2cb-b88a26b5c5b3\") " pod="openshift-dns/node-resolver-cj75q"
Sep 11 12:14:31.027184 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.027162    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltgnt\" (UniqueName: \"kubernetes.io/projected/26ca688b-efaa-42de-b332-78df0daf026b-kube-api-access-ltgnt\") pod \"network-metrics-daemon-bdr64\" (UID: \"26ca688b-efaa-42de-b332-78df0daf026b\") " pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:31.027304 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.027277    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9rvs\" (UniqueName: \"kubernetes.io/projected/455e42df-85a6-4c25-a449-e12375d132e6-kube-api-access-w9rvs\") pod \"node-ca-5bxb8\" (UID: \"455e42df-85a6-4c25-a449-e12375d132e6\") " pod="openshift-image-registry/node-ca-5bxb8"
Sep 11 12:14:31.096038 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.096010    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:31.105200 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.105165    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vw24w"
Sep 11 12:14:31.112410 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.112387    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-trl89"
Sep 11 12:14:31.118279 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.118258    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k7m7m"
Sep 11 12:14:31.124083 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.124061    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rzwzp"
Sep 11 12:14:31.130978 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.130956    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p"
Sep 11 12:14:31.136298 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.136276    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zbbqf"
Sep 11 12:14:31.142880 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.142863    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cj75q"
Sep 11 12:14:31.148009 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.147991    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5bxb8"
Sep 11 12:14:31.411104 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:31.411075    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15eb8bc9_5975_4611_8543_1886caaf3a2f.slice/crio-2d39d6064a29bdbb3d69137dd321dcb19a957b3d18bca83aaddf6bbb3ec22ed7 WatchSource:0}: Error finding container 2d39d6064a29bdbb3d69137dd321dcb19a957b3d18bca83aaddf6bbb3ec22ed7: Status 404 returned error can't find the container with id 2d39d6064a29bdbb3d69137dd321dcb19a957b3d18bca83aaddf6bbb3ec22ed7
Sep 11 12:14:31.412599 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:31.412576    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88402687_4e5b_411c_be7a_4cb955f90425.slice/crio-05c371ac537439b20fe1d3c821aa5f918fe929627be207f073c752d8f0cc2bdb WatchSource:0}: Error finding container 05c371ac537439b20fe1d3c821aa5f918fe929627be207f073c752d8f0cc2bdb: Status 404 returned error can't find the container with id 05c371ac537439b20fe1d3c821aa5f918fe929627be207f073c752d8f0cc2bdb
Sep 11 12:14:31.413984 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:31.413959    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa2dd34_9727_4afd_aae4_066d9cf11e43.slice/crio-6d2dda5f88bcffe3c53a51ba2af0b8a80fd17ab1f70c157841faf644b763fc79 WatchSource:0}: Error finding container 6d2dda5f88bcffe3c53a51ba2af0b8a80fd17ab1f70c157841faf644b763fc79: Status 404 returned error can't find the container with id 6d2dda5f88bcffe3c53a51ba2af0b8a80fd17ab1f70c157841faf644b763fc79
Sep 11 12:14:31.415109 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:31.414541    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a06bf44_c6a4_45cf_a7cb_deefbbf69d2f.slice/crio-9ffbb2caf03be4f946d86b0ac0179697147ef4c10e94d52282d5f5c5315b00bb WatchSource:0}: Error finding container 9ffbb2caf03be4f946d86b0ac0179697147ef4c10e94d52282d5f5c5315b00bb: Status 404 returned error can't find the container with id 9ffbb2caf03be4f946d86b0ac0179697147ef4c10e94d52282d5f5c5315b00bb
Sep 11 12:14:31.416968 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:31.416844    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711fdd9a_1bba_4eac_a618_f4863c741432.slice/crio-fd9ba47159c4eb4a1bf8197c8245ef4b1fc4a9580e92a303fcda958a39d989be WatchSource:0}: Error finding container fd9ba47159c4eb4a1bf8197c8245ef4b1fc4a9580e92a303fcda958a39d989be: Status 404 returned error can't find the container with id fd9ba47159c4eb4a1bf8197c8245ef4b1fc4a9580e92a303fcda958a39d989be
Sep 11 12:14:31.417688 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:31.417672    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5df1e85_4605_4418_a2cb_b88a26b5c5b3.slice/crio-0f5ab13319fef4816e53a4bdcece0f14d23a027591bb5f39d669ec1a29348fcc WatchSource:0}: Error finding container 0f5ab13319fef4816e53a4bdcece0f14d23a027591bb5f39d669ec1a29348fcc: Status 404 returned error can't find the container with id 0f5ab13319fef4816e53a4bdcece0f14d23a027591bb5f39d669ec1a29348fcc
Sep 11 12:14:31.419217 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:31.419094    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c2ca1a_9bbb_4cb5_a226_69eb4db972a2.slice/crio-da74ee240e7580f2ee64cd6d1899e6db9b816b01fb30e1a9390f8435023f4789 WatchSource:0}: Error finding container da74ee240e7580f2ee64cd6d1899e6db9b816b01fb30e1a9390f8435023f4789: Status 404 returned error can't find the container with id da74ee240e7580f2ee64cd6d1899e6db9b816b01fb30e1a9390f8435023f4789
Sep 11 12:14:31.420529 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:31.420315    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5306d10e_12c5_4a0b_b048_c24edb6eed1a.slice/crio-9a552ee6934c94423608e6514a114dde57a199d8fe49b73b13f8834addb4a818 WatchSource:0}: Error finding container 9a552ee6934c94423608e6514a114dde57a199d8fe49b73b13f8834addb4a818: Status 404 returned error can't find the container with id 9a552ee6934c94423608e6514a114dde57a199d8fe49b73b13f8834addb4a818
Sep 11 12:14:31.420770 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:14:31.420752    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455e42df_85a6_4c25_a449_e12375d132e6.slice/crio-1c3a938194be81b1bbbb3f250ab8778c3c12a3feaeee8c249cb702d8e291638b WatchSource:0}: Error finding container 1c3a938194be81b1bbbb3f250ab8778c3c12a3feaeee8c249cb702d8e291638b: Status 404 returned error can't find the container with id 1c3a938194be81b1bbbb3f250ab8778c3c12a3feaeee8c249cb702d8e291638b
Sep 11 12:14:31.517704 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.517682    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2r6g\" (UniqueName: \"kubernetes.io/projected/719c0ee1-cbea-489a-9591-dcfb3e68b4e1-kube-api-access-r2r6g\") pod \"network-check-target-kjxhz\" (UID: \"719c0ee1-cbea-489a-9591-dcfb3e68b4e1\") " pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:14:31.517884 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.517721    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b254d6a2-6adc-468d-b21c-fb5ca9232417-original-pull-secret\") pod \"global-pull-secret-syncer-6qbjc\" (UID: \"b254d6a2-6adc-468d-b21c-fb5ca9232417\") " pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:31.517884 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.517752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ca688b-efaa-42de-b332-78df0daf026b-metrics-certs\") pod \"network-metrics-daemon-bdr64\" (UID: \"26ca688b-efaa-42de-b332-78df0daf026b\") " pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:31.518006 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:31.517906    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:31.518684 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:31.518154    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 12:14:31.518684 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:31.518208    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 12:14:31.518684 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:31.518223    2561 projected.go:196] Error preparing data for projected volume kube-api-access-r2r6g for pod openshift-network-diagnostics/network-check-target-kjxhz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:31.518684 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:31.518257    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b254d6a2-6adc-468d-b21c-fb5ca9232417-original-pull-secret podName:b254d6a2-6adc-468d-b21c-fb5ca9232417 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:32.51823302 +0000 UTC m=+4.081012768 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b254d6a2-6adc-468d-b21c-fb5ca9232417-original-pull-secret") pod "global-pull-secret-syncer-6qbjc" (UID: "b254d6a2-6adc-468d-b21c-fb5ca9232417") : object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:31.518684 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:31.518290    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/719c0ee1-cbea-489a-9591-dcfb3e68b4e1-kube-api-access-r2r6g podName:719c0ee1-cbea-489a-9591-dcfb3e68b4e1 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:32.518280813 +0000 UTC m=+4.081060558 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r2r6g" (UniqueName: "kubernetes.io/projected/719c0ee1-cbea-489a-9591-dcfb3e68b4e1-kube-api-access-r2r6g") pod "network-check-target-kjxhz" (UID: "719c0ee1-cbea-489a-9591-dcfb3e68b4e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:31.518684 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:31.518415    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:31.518684 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:31.518495    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ca688b-efaa-42de-b332-78df0daf026b-metrics-certs podName:26ca688b-efaa-42de-b332-78df0daf026b nodeName:}" failed. No retries permitted until 2026-09-11 12:14:32.518477528 +0000 UTC m=+4.081257271 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26ca688b-efaa-42de-b332-78df0daf026b-metrics-certs") pod "network-metrics-daemon-bdr64" (UID: "26ca688b-efaa-42de-b332-78df0daf026b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:31.864015 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.863906    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 12:09:30 +0000 UTC" deadline="2028-02-10 18:49:03.472025355 +0000 UTC"
Sep 11 12:14:31.864015 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.863963    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12414h34m31.608066444s"
Sep 11 12:14:31.926060 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.926030    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:14:31.926222 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:31.926153    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjxhz" podUID="719c0ee1-cbea-489a-9591-dcfb3e68b4e1"
Sep 11 12:14:31.947467 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.947400    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.ec2.internal" event={"ID":"4d587c23d1130ab850ab1128833ea53c","Type":"ContainerStarted","Data":"10af02bb89cc8d05c3c0825d9c8595ede119b407daced4cf04cb3fb88957deb7"}
Sep 11 12:14:31.950170 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.950109    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5bxb8" event={"ID":"455e42df-85a6-4c25-a449-e12375d132e6","Type":"ContainerStarted","Data":"1c3a938194be81b1bbbb3f250ab8778c3c12a3feaeee8c249cb702d8e291638b"}
Sep 11 12:14:31.951957 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.951899    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zbbqf" event={"ID":"28c2ca1a-9bbb-4cb5-a226-69eb4db972a2","Type":"ContainerStarted","Data":"da74ee240e7580f2ee64cd6d1899e6db9b816b01fb30e1a9390f8435023f4789"}
Sep 11 12:14:31.956111 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.956044    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rzwzp" event={"ID":"5306d10e-12c5-4a0b-b048-c24edb6eed1a","Type":"ContainerStarted","Data":"9a552ee6934c94423608e6514a114dde57a199d8fe49b73b13f8834addb4a818"}
Sep 11 12:14:31.963477 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.963368    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6" event={"ID":"88402687-4e5b-411c-be7a-4cb955f90425","Type":"ContainerStarted","Data":"05c371ac537439b20fe1d3c821aa5f918fe929627be207f073c752d8f0cc2bdb"}
Sep 11 12:14:31.967094 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.967025    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k7m7m" event={"ID":"15eb8bc9-5975-4611-8543-1886caaf3a2f","Type":"ContainerStarted","Data":"2d39d6064a29bdbb3d69137dd321dcb19a957b3d18bca83aaddf6bbb3ec22ed7"}
Sep 11 12:14:31.970224 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.970032    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cj75q" event={"ID":"b5df1e85-4605-4418-a2cb-b88a26b5c5b3","Type":"ContainerStarted","Data":"0f5ab13319fef4816e53a4bdcece0f14d23a027591bb5f39d669ec1a29348fcc"}
Sep 11 12:14:31.974281 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.974209    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-trl89" event={"ID":"711fdd9a-1bba-4eac-a618-f4863c741432","Type":"ContainerStarted","Data":"fd9ba47159c4eb4a1bf8197c8245ef4b1fc4a9580e92a303fcda958a39d989be"}
Sep 11 12:14:31.980746 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.980500    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vw24w" event={"ID":"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f","Type":"ContainerStarted","Data":"9ffbb2caf03be4f946d86b0ac0179697147ef4c10e94d52282d5f5c5315b00bb"}
Sep 11 12:14:31.984633 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:31.984574    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p" event={"ID":"7aa2dd34-9727-4afd-aae4-066d9cf11e43","Type":"ContainerStarted","Data":"6d2dda5f88bcffe3c53a51ba2af0b8a80fd17ab1f70c157841faf644b763fc79"}
Sep 11 12:14:32.526773 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:32.525896    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b254d6a2-6adc-468d-b21c-fb5ca9232417-original-pull-secret\") pod \"global-pull-secret-syncer-6qbjc\" (UID: \"b254d6a2-6adc-468d-b21c-fb5ca9232417\") " pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:32.526773 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:32.525952    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ca688b-efaa-42de-b332-78df0daf026b-metrics-certs\") pod \"network-metrics-daemon-bdr64\" (UID: \"26ca688b-efaa-42de-b332-78df0daf026b\") " pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:32.526773 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:32.526004    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2r6g\" (UniqueName: \"kubernetes.io/projected/719c0ee1-cbea-489a-9591-dcfb3e68b4e1-kube-api-access-r2r6g\") pod \"network-check-target-kjxhz\" (UID: \"719c0ee1-cbea-489a-9591-dcfb3e68b4e1\") " pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:14:32.526773 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:32.526140    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 12:14:32.526773 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:32.526156    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 12:14:32.526773 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:32.526168    2561 projected.go:196] Error preparing data for projected volume kube-api-access-r2r6g for pod openshift-network-diagnostics/network-check-target-kjxhz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:32.526773 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:32.526238    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/719c0ee1-cbea-489a-9591-dcfb3e68b4e1-kube-api-access-r2r6g podName:719c0ee1-cbea-489a-9591-dcfb3e68b4e1 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:34.526221037 +0000 UTC m=+6.089000784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r2r6g" (UniqueName: "kubernetes.io/projected/719c0ee1-cbea-489a-9591-dcfb3e68b4e1-kube-api-access-r2r6g") pod "network-check-target-kjxhz" (UID: "719c0ee1-cbea-489a-9591-dcfb3e68b4e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:32.526773 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:32.526313    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:32.526773 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:32.526349    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b254d6a2-6adc-468d-b21c-fb5ca9232417-original-pull-secret podName:b254d6a2-6adc-468d-b21c-fb5ca9232417 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:34.526337905 +0000 UTC m=+6.089117649 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b254d6a2-6adc-468d-b21c-fb5ca9232417-original-pull-secret") pod "global-pull-secret-syncer-6qbjc" (UID: "b254d6a2-6adc-468d-b21c-fb5ca9232417") : object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:32.526773 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:32.526408    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:32.526773 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:32.526438    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ca688b-efaa-42de-b332-78df0daf026b-metrics-certs podName:26ca688b-efaa-42de-b332-78df0daf026b nodeName:}" failed. No retries permitted until 2026-09-11 12:14:34.526427563 +0000 UTC m=+6.089207309 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26ca688b-efaa-42de-b332-78df0daf026b-metrics-certs") pod "network-metrics-daemon-bdr64" (UID: "26ca688b-efaa-42de-b332-78df0daf026b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:32.864252 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:32.864143    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 12:09:30 +0000 UTC" deadline="2028-04-17 08:52:24.497508312 +0000 UTC"
Sep 11 12:14:32.864252 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:32.864201    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14012h37m51.63332573s"
Sep 11 12:14:32.926323 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:32.926298    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:32.926467 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:32.926423    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qbjc" podUID="b254d6a2-6adc-468d-b21c-fb5ca9232417"
Sep 11 12:14:32.926910 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:32.926889    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:32.927028 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:32.927007    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdr64" podUID="26ca688b-efaa-42de-b332-78df0daf026b"
Sep 11 12:14:33.926286 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:33.926252    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:14:33.926740 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:33.926395    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjxhz" podUID="719c0ee1-cbea-489a-9591-dcfb3e68b4e1"
Sep 11 12:14:34.542543 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:34.542508    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2r6g\" (UniqueName: \"kubernetes.io/projected/719c0ee1-cbea-489a-9591-dcfb3e68b4e1-kube-api-access-r2r6g\") pod \"network-check-target-kjxhz\" (UID: \"719c0ee1-cbea-489a-9591-dcfb3e68b4e1\") " pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:14:34.542733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:34.542605    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b254d6a2-6adc-468d-b21c-fb5ca9232417-original-pull-secret\") pod \"global-pull-secret-syncer-6qbjc\" (UID: \"b254d6a2-6adc-468d-b21c-fb5ca9232417\") " pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:34.542733 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:34.542645    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 12:14:34.542733 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:34.542665    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 12:14:34.542733 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:34.542677    2561 projected.go:196] Error preparing data for projected volume kube-api-access-r2r6g for pod openshift-network-diagnostics/network-check-target-kjxhz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:34.542733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:34.542681    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ca688b-efaa-42de-b332-78df0daf026b-metrics-certs\") pod \"network-metrics-daemon-bdr64\" (UID: \"26ca688b-efaa-42de-b332-78df0daf026b\") " pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:34.542733 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:34.542720    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/719c0ee1-cbea-489a-9591-dcfb3e68b4e1-kube-api-access-r2r6g podName:719c0ee1-cbea-489a-9591-dcfb3e68b4e1 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:38.542707716 +0000 UTC m=+10.105487456 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r2r6g" (UniqueName: "kubernetes.io/projected/719c0ee1-cbea-489a-9591-dcfb3e68b4e1-kube-api-access-r2r6g") pod "network-check-target-kjxhz" (UID: "719c0ee1-cbea-489a-9591-dcfb3e68b4e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:34.543052 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:34.542839    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:34.543052 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:34.542889    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ca688b-efaa-42de-b332-78df0daf026b-metrics-certs podName:26ca688b-efaa-42de-b332-78df0daf026b nodeName:}" failed. No retries permitted until 2026-09-11 12:14:38.542874223 +0000 UTC m=+10.105653966 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26ca688b-efaa-42de-b332-78df0daf026b-metrics-certs") pod "network-metrics-daemon-bdr64" (UID: "26ca688b-efaa-42de-b332-78df0daf026b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:34.543052 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:34.542947    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:34.543052 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:34.542984    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b254d6a2-6adc-468d-b21c-fb5ca9232417-original-pull-secret podName:b254d6a2-6adc-468d-b21c-fb5ca9232417 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:38.542972939 +0000 UTC m=+10.105752687 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b254d6a2-6adc-468d-b21c-fb5ca9232417-original-pull-secret") pod "global-pull-secret-syncer-6qbjc" (UID: "b254d6a2-6adc-468d-b21c-fb5ca9232417") : object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:34.928076 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:34.928000    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:34.928537 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:34.928138    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qbjc" podUID="b254d6a2-6adc-468d-b21c-fb5ca9232417"
Sep 11 12:14:34.928537 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:34.928512    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:34.928651 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:34.928612    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdr64" podUID="26ca688b-efaa-42de-b332-78df0daf026b"
Sep 11 12:14:35.926802 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:35.926759    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:14:35.926997 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:35.926892    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjxhz" podUID="719c0ee1-cbea-489a-9591-dcfb3e68b4e1"
Sep 11 12:14:36.926930 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:36.926901    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:36.927388 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:36.927038    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qbjc" podUID="b254d6a2-6adc-468d-b21c-fb5ca9232417"
Sep 11 12:14:36.927388 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:36.927072    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:36.927388 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:36.927153    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdr64" podUID="26ca688b-efaa-42de-b332-78df0daf026b"
Sep 11 12:14:37.926198 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:37.926107    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:14:37.926366 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:37.926252    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjxhz" podUID="719c0ee1-cbea-489a-9591-dcfb3e68b4e1"
Sep 11 12:14:38.574743 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:38.574703    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2r6g\" (UniqueName: \"kubernetes.io/projected/719c0ee1-cbea-489a-9591-dcfb3e68b4e1-kube-api-access-r2r6g\") pod \"network-check-target-kjxhz\" (UID: \"719c0ee1-cbea-489a-9591-dcfb3e68b4e1\") " pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:14:38.575207 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:38.574769    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b254d6a2-6adc-468d-b21c-fb5ca9232417-original-pull-secret\") pod \"global-pull-secret-syncer-6qbjc\" (UID: \"b254d6a2-6adc-468d-b21c-fb5ca9232417\") " pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:38.575207 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:38.574811    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ca688b-efaa-42de-b332-78df0daf026b-metrics-certs\") pod \"network-metrics-daemon-bdr64\" (UID: \"26ca688b-efaa-42de-b332-78df0daf026b\") " pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:38.575207 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:38.574883    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 12:14:38.575207 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:38.574904    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 12:14:38.575207 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:38.574911    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:38.575207 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:38.574916    2561 projected.go:196] Error preparing data for projected volume kube-api-access-r2r6g for pod openshift-network-diagnostics/network-check-target-kjxhz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:38.575207 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:38.574964    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/719c0ee1-cbea-489a-9591-dcfb3e68b4e1-kube-api-access-r2r6g podName:719c0ee1-cbea-489a-9591-dcfb3e68b4e1 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:46.574946537 +0000 UTC m=+18.137726283 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r2r6g" (UniqueName: "kubernetes.io/projected/719c0ee1-cbea-489a-9591-dcfb3e68b4e1-kube-api-access-r2r6g") pod "network-check-target-kjxhz" (UID: "719c0ee1-cbea-489a-9591-dcfb3e68b4e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:38.575207 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:38.574980    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:38.575207 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:38.574985    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ca688b-efaa-42de-b332-78df0daf026b-metrics-certs podName:26ca688b-efaa-42de-b332-78df0daf026b nodeName:}" failed. No retries permitted until 2026-09-11 12:14:46.574974912 +0000 UTC m=+18.137754659 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26ca688b-efaa-42de-b332-78df0daf026b-metrics-certs") pod "network-metrics-daemon-bdr64" (UID: "26ca688b-efaa-42de-b332-78df0daf026b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:38.575207 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:38.575010    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b254d6a2-6adc-468d-b21c-fb5ca9232417-original-pull-secret podName:b254d6a2-6adc-468d-b21c-fb5ca9232417 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:46.575000876 +0000 UTC m=+18.137780617 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b254d6a2-6adc-468d-b21c-fb5ca9232417-original-pull-secret") pod "global-pull-secret-syncer-6qbjc" (UID: "b254d6a2-6adc-468d-b21c-fb5ca9232417") : object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:38.927261 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:38.926739    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:38.927261 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:38.926871    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qbjc" podUID="b254d6a2-6adc-468d-b21c-fb5ca9232417"
Sep 11 12:14:38.927474 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:38.927334    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:38.927474 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:38.927445    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdr64" podUID="26ca688b-efaa-42de-b332-78df0daf026b"
Sep 11 12:14:39.925901 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:39.925871    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:14:39.926365 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:39.925980    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjxhz" podUID="719c0ee1-cbea-489a-9591-dcfb3e68b4e1"
Sep 11 12:14:40.926525 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:40.926486    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:40.926966 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:40.926486    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:40.926966 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:40.926644    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdr64" podUID="26ca688b-efaa-42de-b332-78df0daf026b"
Sep 11 12:14:40.926966 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:40.926663    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qbjc" podUID="b254d6a2-6adc-468d-b21c-fb5ca9232417"
Sep 11 12:14:41.926173 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:41.926093    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:14:41.926407 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:41.926225    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjxhz" podUID="719c0ee1-cbea-489a-9591-dcfb3e68b4e1"
Sep 11 12:14:42.928660 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:42.928629    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:42.929105 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:42.928668    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:42.929105 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:42.928776    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdr64" podUID="26ca688b-efaa-42de-b332-78df0daf026b"
Sep 11 12:14:42.929105 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:42.928871    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qbjc" podUID="b254d6a2-6adc-468d-b21c-fb5ca9232417"
Sep 11 12:14:43.926511 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:43.926476    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:14:43.926701 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:43.926598    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjxhz" podUID="719c0ee1-cbea-489a-9591-dcfb3e68b4e1"
Sep 11 12:14:44.926898 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:44.926862    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:44.927440 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:44.926968    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qbjc" podUID="b254d6a2-6adc-468d-b21c-fb5ca9232417"
Sep 11 12:14:44.927440 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:44.927070    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:44.927440 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:44.927203    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdr64" podUID="26ca688b-efaa-42de-b332-78df0daf026b"
Sep 11 12:14:45.926295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:45.926264    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:14:45.926566 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:45.926381    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjxhz" podUID="719c0ee1-cbea-489a-9591-dcfb3e68b4e1"
Sep 11 12:14:46.632868 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:46.632834    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2r6g\" (UniqueName: \"kubernetes.io/projected/719c0ee1-cbea-489a-9591-dcfb3e68b4e1-kube-api-access-r2r6g\") pod \"network-check-target-kjxhz\" (UID: \"719c0ee1-cbea-489a-9591-dcfb3e68b4e1\") " pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:14:46.633304 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:46.632876    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b254d6a2-6adc-468d-b21c-fb5ca9232417-original-pull-secret\") pod \"global-pull-secret-syncer-6qbjc\" (UID: \"b254d6a2-6adc-468d-b21c-fb5ca9232417\") " pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:46.633304 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:46.632911    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ca688b-efaa-42de-b332-78df0daf026b-metrics-certs\") pod \"network-metrics-daemon-bdr64\" (UID: \"26ca688b-efaa-42de-b332-78df0daf026b\") " pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:46.633304 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:46.633079    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:46.633304 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:46.633081    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:46.633304 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:46.633129    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ca688b-efaa-42de-b332-78df0daf026b-metrics-certs podName:26ca688b-efaa-42de-b332-78df0daf026b nodeName:}" failed. No retries permitted until 2026-09-11 12:15:02.633111363 +0000 UTC m=+34.195891104 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26ca688b-efaa-42de-b332-78df0daf026b-metrics-certs") pod "network-metrics-daemon-bdr64" (UID: "26ca688b-efaa-42de-b332-78df0daf026b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:14:46.633304 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:46.633082    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 12:14:46.633304 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:46.633150    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b254d6a2-6adc-468d-b21c-fb5ca9232417-original-pull-secret podName:b254d6a2-6adc-468d-b21c-fb5ca9232417 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:02.633139804 +0000 UTC m=+34.195919549 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b254d6a2-6adc-468d-b21c-fb5ca9232417-original-pull-secret") pod "global-pull-secret-syncer-6qbjc" (UID: "b254d6a2-6adc-468d-b21c-fb5ca9232417") : object "kube-system"/"original-pull-secret" not registered
Sep 11 12:14:46.633304 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:46.633156    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 12:14:46.633304 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:46.633170    2561 projected.go:196] Error preparing data for projected volume kube-api-access-r2r6g for pod openshift-network-diagnostics/network-check-target-kjxhz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:46.633304 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:46.633250    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/719c0ee1-cbea-489a-9591-dcfb3e68b4e1-kube-api-access-r2r6g podName:719c0ee1-cbea-489a-9591-dcfb3e68b4e1 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:02.633222523 +0000 UTC m=+34.196002270 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r2r6g" (UniqueName: "kubernetes.io/projected/719c0ee1-cbea-489a-9591-dcfb3e68b4e1-kube-api-access-r2r6g") pod "network-check-target-kjxhz" (UID: "719c0ee1-cbea-489a-9591-dcfb3e68b4e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:14:46.925918 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:46.925837    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:46.926070 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:46.925969    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qbjc" podUID="b254d6a2-6adc-468d-b21c-fb5ca9232417"
Sep 11 12:14:46.926070 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:46.926009    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:46.926233 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:46.926164    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdr64" podUID="26ca688b-efaa-42de-b332-78df0daf026b"
Sep 11 12:14:47.926744 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:47.926714    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:14:47.927115 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:47.926812    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjxhz" podUID="719c0ee1-cbea-489a-9591-dcfb3e68b4e1"
Sep 11 12:14:48.926970 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:48.926941    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:48.927360 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:48.927018    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qbjc" podUID="b254d6a2-6adc-468d-b21c-fb5ca9232417"
Sep 11 12:14:48.927360 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:48.927116    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:48.927360 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:48.927233    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdr64" podUID="26ca688b-efaa-42de-b332-78df0daf026b"
Sep 11 12:14:49.926451 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:49.926088    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:14:49.926582 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:49.926554    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjxhz" podUID="719c0ee1-cbea-489a-9591-dcfb3e68b4e1"
Sep 11 12:14:50.035920 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.035888    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5bxb8" event={"ID":"455e42df-85a6-4c25-a449-e12375d132e6","Type":"ContainerStarted","Data":"7c2b56330c098fa28d38ea934eb684db19c5054d1dc3f305e8f96e9a0f19734f"}
Sep 11 12:14:50.037511 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.037481    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zbbqf" event={"ID":"28c2ca1a-9bbb-4cb5-a226-69eb4db972a2","Type":"ContainerStarted","Data":"e6bcd782f51b6efbce3ec3cd8be579567f9ecc5b6a93baa32e3108e6a5ebcc12"}
Sep 11 12:14:50.040273 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.040255    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcfm6_88402687-4e5b-411c-be7a-4cb955f90425/ovn-acl-logging/0.log"
Sep 11 12:14:50.040613 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.040595    2561 generic.go:356] "Generic (PLEG): container finished" podID="88402687-4e5b-411c-be7a-4cb955f90425" containerID="92017a06ffa42b081a25ebad7dbefc91af21d31ef5d56bf9a9eea4b1b442533b" exitCode=1
Sep 11 12:14:50.040682 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.040664    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6" event={"ID":"88402687-4e5b-411c-be7a-4cb955f90425","Type":"ContainerStarted","Data":"4871cd48e24c1438436c9f1dfd944480701211d48d7e061df9a078b6059ac2c9"}
Sep 11 12:14:50.040755 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.040725    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6" event={"ID":"88402687-4e5b-411c-be7a-4cb955f90425","Type":"ContainerStarted","Data":"14e3344c53d6c1d81108b6f482ccbf3bd2e538a8144ce4981d451ff6832ab594"}
Sep 11 12:14:50.040817 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.040758    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6" event={"ID":"88402687-4e5b-411c-be7a-4cb955f90425","Type":"ContainerStarted","Data":"f0b18c9ce5698743064912049e767aa3caa965a6ee08f09ed0af37547ae70b03"}
Sep 11 12:14:50.040817 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.040785    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6" event={"ID":"88402687-4e5b-411c-be7a-4cb955f90425","Type":"ContainerStarted","Data":"8c55a2c674fe65d6361dd77e011affea062bfc70d8d687c9e9cfc681c00309f5"}
Sep 11 12:14:50.040904 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.040820    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6" event={"ID":"88402687-4e5b-411c-be7a-4cb955f90425","Type":"ContainerDied","Data":"92017a06ffa42b081a25ebad7dbefc91af21d31ef5d56bf9a9eea4b1b442533b"}
Sep 11 12:14:50.040904 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.040856    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6" event={"ID":"88402687-4e5b-411c-be7a-4cb955f90425","Type":"ContainerStarted","Data":"2894b4fed540a016e84927163f70afe72b0ba2e780842d645fb1aae5e28e9340"}
Sep 11 12:14:50.042054 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.042033    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k7m7m" event={"ID":"15eb8bc9-5975-4611-8543-1886caaf3a2f","Type":"ContainerStarted","Data":"240deb2e3eb71cb500ec948a341743fdbb4eed2d2945db3a8642d05a8fde9c97"}
Sep 11 12:14:50.043402 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.043385    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cj75q" event={"ID":"b5df1e85-4605-4418-a2cb-b88a26b5c5b3","Type":"ContainerStarted","Data":"8663f221cf0421f5895ed20068544d24302e4a7020fabdb850ef8b79e677ef64"}
Sep 11 12:14:50.044713 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.044692    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-trl89" event={"ID":"711fdd9a-1bba-4eac-a618-f4863c741432","Type":"ContainerStarted","Data":"0ef324df01085d632b99e009577111a3af8be5d5f37670214382ed5ae6980132"}
Sep 11 12:14:50.046168 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.046148    2561 generic.go:356] "Generic (PLEG): container finished" podID="0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f" containerID="bed6c17c128ae7b750885cad14695216d49bb0cc4c7347216134ee8c51555a04" exitCode=0
Sep 11 12:14:50.046271 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.046211    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vw24w" event={"ID":"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f","Type":"ContainerDied","Data":"bed6c17c128ae7b750885cad14695216d49bb0cc4c7347216134ee8c51555a04"}
Sep 11 12:14:50.047762 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.047746    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p" event={"ID":"7aa2dd34-9727-4afd-aae4-066d9cf11e43","Type":"ContainerStarted","Data":"613b5b3dacc4d4d4c058cebed8f9b814aaf68ce2365020398f673e7f73f0559f"}
Sep 11 12:14:50.049308 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.049285    2561 generic.go:356] "Generic (PLEG): container finished" podID="916ee7326c953598ae16ef7c8e25eae2" containerID="7352536da03b5bb6ee2bf25eae0cacc375c5a8ca8b5e6f68f73e2b22d135e6dc" exitCode=0
Sep 11 12:14:50.049406 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.049316    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.ec2.internal" event={"ID":"916ee7326c953598ae16ef7c8e25eae2","Type":"ContainerDied","Data":"7352536da03b5bb6ee2bf25eae0cacc375c5a8ca8b5e6f68f73e2b22d135e6dc"}
Sep 11 12:14:50.052292 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.052253    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.ec2.internal" podStartSLOduration=20.052242089 podStartE2EDuration="20.052242089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:31.971628794 +0000 UTC m=+3.534408558" watchObservedRunningTime="2026-09-11 12:14:50.052242089 +0000 UTC m=+21.615021851"
Sep 11 12:14:50.052420 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.052402    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5bxb8" podStartSLOduration=7.524697191 podStartE2EDuration="21.052396892s" totalImagesPullingTime="13.527699701s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:50.052145449 +0000 UTC m=+21.614925213" watchObservedRunningTime="2026-09-11 12:14:50.052396892 +0000 UTC m=+21.615176655"
Sep 11 12:14:50.094134 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.094087    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-cj75q" podStartSLOduration=3.453848647 podStartE2EDuration="21.094068747s" totalImagesPullingTime="17.6402201s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:50.093135542 +0000 UTC m=+21.655915307" watchObservedRunningTime="2026-09-11 12:14:50.094068747 +0000 UTC m=+21.656848512"
Sep 11 12:14:50.130387 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.130343    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-trl89" podStartSLOduration=3.487288417 podStartE2EDuration="21.130333815s" totalImagesPullingTime="17.643045398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:50.130297923 +0000 UTC m=+21.693077686" watchObservedRunningTime="2026-09-11 12:14:50.130333815 +0000 UTC m=+21.693113577"
Sep 11 12:14:50.147899 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.147861    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-k7m7m" podStartSLOduration=3.106428104 podStartE2EDuration="21.147846899s" totalImagesPullingTime="18.041418795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:50.147661677 +0000 UTC m=+21.710441440" watchObservedRunningTime="2026-09-11 12:14:50.147846899 +0000 UTC m=+21.710626662"
Sep 11 12:14:50.166873 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.166822    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zbbqf" podStartSLOduration=7.63880666 podStartE2EDuration="21.166803853s" totalImagesPullingTime="13.527997193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:50.16628668 +0000 UTC m=+21.729066442" watchObservedRunningTime="2026-09-11 12:14:50.166803853 +0000 UTC m=+21.729583617"
Sep 11 12:14:50.814481 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.814453    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 12:14:50.856319 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.856201    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T12:14:50.814477176Z","UUID":"983542c6-e79b-450e-b0fa-783624344806","Handler":null,"Name":"","Endpoint":""}
Sep 11 12:14:50.857787 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.857766    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 12:14:50.857904 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.857796    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 12:14:50.926888 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.926860    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:50.927012 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:50.926982    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qbjc" podUID="b254d6a2-6adc-468d-b21c-fb5ca9232417"
Sep 11 12:14:50.927074 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:50.927040    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:50.927148 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:50.927130    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdr64" podUID="26ca688b-efaa-42de-b332-78df0daf026b"
Sep 11 12:14:51.053701 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:51.053658    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.ec2.internal" event={"ID":"916ee7326c953598ae16ef7c8e25eae2","Type":"ContainerStarted","Data":"344943a8ff8e1f7b5f21b90cbb1f526747e0020d876a8e5162b4520e9d9159c3"}
Sep 11 12:14:51.055454 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:51.055426    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rzwzp" event={"ID":"5306d10e-12c5-4a0b-b048-c24edb6eed1a","Type":"ContainerStarted","Data":"2e5d7fe6b32203092ae96e1b966dfb0ada651b8ce7dafac7f96bfe7fb42fd308"}
Sep 11 12:14:51.057507 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:51.057438    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p" event={"ID":"7aa2dd34-9727-4afd-aae4-066d9cf11e43","Type":"ContainerStarted","Data":"dd60fbeaca8abcd2be5b89baf49ab6d958726ed28b4e390830fec21a452b2bac"}
Sep 11 12:14:51.071129 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:51.071089    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.ec2.internal" podStartSLOduration=21.071077823 podStartE2EDuration="21.071077823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:51.070538618 +0000 UTC m=+22.633318388" watchObservedRunningTime="2026-09-11 12:14:51.071077823 +0000 UTC m=+22.633857586"
Sep 11 12:14:51.086100 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:51.086061    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rzwzp" podStartSLOduration=4.461179139 podStartE2EDuration="22.086046354s" totalImagesPullingTime="17.624867215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:51.085688779 +0000 UTC m=+22.648468543" watchObservedRunningTime="2026-09-11 12:14:51.086046354 +0000 UTC m=+22.648826120"
Sep 11 12:14:51.926433 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:51.926184    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:14:51.926602 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:51.926549    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjxhz" podUID="719c0ee1-cbea-489a-9591-dcfb3e68b4e1"
Sep 11 12:14:52.064736 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:52.064710    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcfm6_88402687-4e5b-411c-be7a-4cb955f90425/ovn-acl-logging/0.log"
Sep 11 12:14:52.065204 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:52.065163    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6" event={"ID":"88402687-4e5b-411c-be7a-4cb955f90425","Type":"ContainerStarted","Data":"6e4ce8536553b647d6e6f34ffb96afcbdc040a93b3bcb625cd1a05a4efd04b0a"}
Sep 11 12:14:52.926261 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:52.926231    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:52.926467 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:52.926352    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qbjc" podUID="b254d6a2-6adc-468d-b21c-fb5ca9232417"
Sep 11 12:14:52.926467 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:52.926391    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:52.926591 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:52.926516    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdr64" podUID="26ca688b-efaa-42de-b332-78df0daf026b"
Sep 11 12:14:53.072139 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:53.072107    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p" event={"ID":"7aa2dd34-9727-4afd-aae4-066d9cf11e43","Type":"ContainerStarted","Data":"bd1266639ec96a1f371583e5dc26d4df74aac39ed984ac11f47e2d1a7ee9e70f"}
Sep 11 12:14:53.093688 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:53.093644    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b7f5p" podStartSLOduration=3.979868688 podStartE2EDuration="24.093627237s" totalImagesPullingTime="20.113758549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:53.093559412 +0000 UTC m=+24.656339175" watchObservedRunningTime="2026-09-11 12:14:53.093627237 +0000 UTC m=+24.656407000"
Sep 11 12:14:53.926830 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:53.926795    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:14:53.926973 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:53.926933    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjxhz" podUID="719c0ee1-cbea-489a-9591-dcfb3e68b4e1"
Sep 11 12:14:54.322988 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:54.322834    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zbbqf"
Sep 11 12:14:54.323476 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:54.323459    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zbbqf"
Sep 11 12:14:54.926812 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:54.926782    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:54.927121 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:54.926877    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qbjc" podUID="b254d6a2-6adc-468d-b21c-fb5ca9232417"
Sep 11 12:14:54.927253 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:54.927236    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:54.927341 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:54.927324    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdr64" podUID="26ca688b-efaa-42de-b332-78df0daf026b"
Sep 11 12:14:55.078713 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:55.078686    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcfm6_88402687-4e5b-411c-be7a-4cb955f90425/ovn-acl-logging/0.log"
Sep 11 12:14:55.079091 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:55.079062    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6" event={"ID":"88402687-4e5b-411c-be7a-4cb955f90425","Type":"ContainerStarted","Data":"b3257459e85871cf642e02118ec0ffd0becf9d97e5eeeb522410a907e478ab61"}
Sep 11 12:14:55.079330 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:55.079307    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:55.079445 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:55.079377    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:55.079538 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:55.079506    2561 scope.go:117] "RemoveContainer" containerID="92017a06ffa42b081a25ebad7dbefc91af21d31ef5d56bf9a9eea4b1b442533b"
Sep 11 12:14:55.081172 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:55.081135    2561 generic.go:356] "Generic (PLEG): container finished" podID="0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f" containerID="41c5a5a243fc1822f73dc44af980931727725de283ce4144581f92c5f0dec38a" exitCode=0
Sep 11 12:14:55.081293 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:55.081169    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vw24w" event={"ID":"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f","Type":"ContainerDied","Data":"41c5a5a243fc1822f73dc44af980931727725de283ce4144581f92c5f0dec38a"}
Sep 11 12:14:55.081491 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:55.081469    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zbbqf"
Sep 11 12:14:55.082013 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:55.081996    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zbbqf"
Sep 11 12:14:55.103349 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:55.103330    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:55.926461 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:55.926430    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:14:55.927070 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:55.926535    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjxhz" podUID="719c0ee1-cbea-489a-9591-dcfb3e68b4e1"
Sep 11 12:14:56.066429 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:56.066371    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bdr64"]
Sep 11 12:14:56.066542 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:56.066507    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:56.066644 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:56.066621    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdr64" podUID="26ca688b-efaa-42de-b332-78df0daf026b"
Sep 11 12:14:56.076302 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:56.076178    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kjxhz"]
Sep 11 12:14:56.077238 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:56.077215    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6qbjc"]
Sep 11 12:14:56.077341 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:56.077324    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:56.077439 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:56.077412    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qbjc" podUID="b254d6a2-6adc-468d-b21c-fb5ca9232417"
Sep 11 12:14:56.088935 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:56.088914    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcfm6_88402687-4e5b-411c-be7a-4cb955f90425/ovn-acl-logging/0.log"
Sep 11 12:14:56.089412 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:56.089377    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6" event={"ID":"88402687-4e5b-411c-be7a-4cb955f90425","Type":"ContainerStarted","Data":"26ae0762eb5c7c3af2c589a7b59ef7f9d18a9782b5c07e6eaaf39c43cb7d10cc"}
Sep 11 12:14:56.089522 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:56.089481    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:14:56.089585 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:56.089523    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:56.089696 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:56.089665    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjxhz" podUID="719c0ee1-cbea-489a-9591-dcfb3e68b4e1"
Sep 11 12:14:56.110991 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:56.110970    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:14:56.123306 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:56.123260    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6" podStartSLOduration=9.456497412 podStartE2EDuration="27.1232427s" totalImagesPullingTime="17.666745288s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:56.121981961 +0000 UTC m=+27.684761723" watchObservedRunningTime="2026-09-11 12:14:56.1232427 +0000 UTC m=+27.686022460"
Sep 11 12:14:57.093607 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:57.093423    2561 generic.go:356] "Generic (PLEG): container finished" podID="0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f" containerID="32eeff601845aa25f62fc512076fd0d4295e2e21ccd3a6efea6c31f0ba9f096a" exitCode=0
Sep 11 12:14:57.094019 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:57.093507    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vw24w" event={"ID":"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f","Type":"ContainerDied","Data":"32eeff601845aa25f62fc512076fd0d4295e2e21ccd3a6efea6c31f0ba9f096a"}
Sep 11 12:14:57.926603 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:57.926570    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:57.926784 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:57.926605    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:57.926784 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:57.926575    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:14:57.926784 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:57.926723    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdr64" podUID="26ca688b-efaa-42de-b332-78df0daf026b"
Sep 11 12:14:57.926938 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:57.926785    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjxhz" podUID="719c0ee1-cbea-489a-9591-dcfb3e68b4e1"
Sep 11 12:14:57.926938 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:57.926839    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qbjc" podUID="b254d6a2-6adc-468d-b21c-fb5ca9232417"
Sep 11 12:14:59.100539 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:59.100503    2561 generic.go:356] "Generic (PLEG): container finished" podID="0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f" containerID="4552dec97904bc72cc394a39be62ed7f5335f372d979ce0095b0fd4b776c2b4b" exitCode=0
Sep 11 12:14:59.100905 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:59.100551    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vw24w" event={"ID":"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f","Type":"ContainerDied","Data":"4552dec97904bc72cc394a39be62ed7f5335f372d979ce0095b0fd4b776c2b4b"}
Sep 11 12:14:59.926054 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:59.926025    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:14:59.926054 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:59.926039    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:14:59.926430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:14:59.926025    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:14:59.926430 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:59.926141    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjxhz" podUID="719c0ee1-cbea-489a-9591-dcfb3e68b4e1"
Sep 11 12:14:59.926430 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:59.926230    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qbjc" podUID="b254d6a2-6adc-468d-b21c-fb5ca9232417"
Sep 11 12:14:59.926430 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:14:59.926318    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdr64" podUID="26ca688b-efaa-42de-b332-78df0daf026b"
Sep 11 12:15:01.926822 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:01.926799    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:15:01.927163 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:01.926800    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:15:01.927163 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:01.926898    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kjxhz" podUID="719c0ee1-cbea-489a-9591-dcfb3e68b4e1"
Sep 11 12:15:01.927163 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:01.926976    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdr64" podUID="26ca688b-efaa-42de-b332-78df0daf026b"
Sep 11 12:15:01.927163 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:01.926800    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:15:01.927163 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:01.927050    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qbjc" podUID="b254d6a2-6adc-468d-b21c-fb5ca9232417"
Sep 11 12:15:02.207953 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.207874    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-172.ec2.internal" event="NodeReady"
Sep 11 12:15:02.208113 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.208016    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 12:15:02.261220 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.261178    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ptkj7"]
Sep 11 12:15:02.281795 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.281769    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-n5nml"]
Sep 11 12:15:02.281957 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.281903    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ptkj7"
Sep 11 12:15:02.284379 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.284342    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 12:15:02.284975 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.284858    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lgchp\""
Sep 11 12:15:02.284975 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.284870    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 12:15:02.295752 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.295731    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ptkj7"]
Sep 11 12:15:02.295855 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.295771    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n5nml"]
Sep 11 12:15:02.295909 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.295858    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n5nml"
Sep 11 12:15:02.298653 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.298631    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 12:15:02.299276 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.299257    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ld655\""
Sep 11 12:15:02.299422 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.299398    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 12:15:02.299504 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.299401    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 12:15:02.450923 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.450872    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25006c65-374d-49ed-b247-a8e0e9833b34-cert\") pod \"ingress-canary-n5nml\" (UID: \"25006c65-374d-49ed-b247-a8e0e9833b34\") " pod="openshift-ingress-canary/ingress-canary-n5nml"
Sep 11 12:15:02.451108 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.450940    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-config-volume\") pod \"dns-default-ptkj7\" (UID: \"4e3ebd74-ae34-4842-8ba8-184dc4cb54cb\") " pod="openshift-dns/dns-default-ptkj7"
Sep 11 12:15:02.451108 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.450965    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-tmp-dir\") pod \"dns-default-ptkj7\" (UID: \"4e3ebd74-ae34-4842-8ba8-184dc4cb54cb\") " pod="openshift-dns/dns-default-ptkj7"
Sep 11 12:15:02.451108 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.450990    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5jsn\" (UniqueName: \"kubernetes.io/projected/25006c65-374d-49ed-b247-a8e0e9833b34-kube-api-access-l5jsn\") pod \"ingress-canary-n5nml\" (UID: \"25006c65-374d-49ed-b247-a8e0e9833b34\") " pod="openshift-ingress-canary/ingress-canary-n5nml"
Sep 11 12:15:02.451108 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.451080    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55hrp\" (UniqueName: \"kubernetes.io/projected/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-kube-api-access-55hrp\") pod \"dns-default-ptkj7\" (UID: \"4e3ebd74-ae34-4842-8ba8-184dc4cb54cb\") " pod="openshift-dns/dns-default-ptkj7"
Sep 11 12:15:02.451108 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.451107    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-metrics-tls\") pod \"dns-default-ptkj7\" (UID: \"4e3ebd74-ae34-4842-8ba8-184dc4cb54cb\") " pod="openshift-dns/dns-default-ptkj7"
Sep 11 12:15:02.552220 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.552166    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-metrics-tls\") pod \"dns-default-ptkj7\" (UID: \"4e3ebd74-ae34-4842-8ba8-184dc4cb54cb\") " pod="openshift-dns/dns-default-ptkj7"
Sep 11 12:15:02.552400 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.552274    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25006c65-374d-49ed-b247-a8e0e9833b34-cert\") pod \"ingress-canary-n5nml\" (UID: \"25006c65-374d-49ed-b247-a8e0e9833b34\") " pod="openshift-ingress-canary/ingress-canary-n5nml"
Sep 11 12:15:02.552400 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.552312    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-config-volume\") pod \"dns-default-ptkj7\" (UID: \"4e3ebd74-ae34-4842-8ba8-184dc4cb54cb\") " pod="openshift-dns/dns-default-ptkj7"
Sep 11 12:15:02.552400 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:02.552322    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 12:15:02.552400 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.552338    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-tmp-dir\") pod \"dns-default-ptkj7\" (UID: \"4e3ebd74-ae34-4842-8ba8-184dc4cb54cb\") " pod="openshift-dns/dns-default-ptkj7"
Sep 11 12:15:02.552400 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.552386    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5jsn\" (UniqueName: \"kubernetes.io/projected/25006c65-374d-49ed-b247-a8e0e9833b34-kube-api-access-l5jsn\") pod \"ingress-canary-n5nml\" (UID: \"25006c65-374d-49ed-b247-a8e0e9833b34\") " pod="openshift-ingress-canary/ingress-canary-n5nml"
Sep 11 12:15:02.552644 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.552430    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55hrp\" (UniqueName: \"kubernetes.io/projected/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-kube-api-access-55hrp\") pod \"dns-default-ptkj7\" (UID: \"4e3ebd74-ae34-4842-8ba8-184dc4cb54cb\") " pod="openshift-dns/dns-default-ptkj7"
Sep 11 12:15:02.552765 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:02.552744    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-metrics-tls podName:4e3ebd74-ae34-4842-8ba8-184dc4cb54cb nodeName:}" failed. No retries permitted until 2026-09-11 12:15:03.052719905 +0000 UTC m=+34.615499653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-metrics-tls") pod "dns-default-ptkj7" (UID: "4e3ebd74-ae34-4842-8ba8-184dc4cb54cb") : secret "dns-default-metrics-tls" not found
Sep 11 12:15:02.552765 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.552756    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-tmp-dir\") pod \"dns-default-ptkj7\" (UID: \"4e3ebd74-ae34-4842-8ba8-184dc4cb54cb\") " pod="openshift-dns/dns-default-ptkj7"
Sep 11 12:15:02.553291 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.553262    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-config-volume\") pod \"dns-default-ptkj7\" (UID: \"4e3ebd74-ae34-4842-8ba8-184dc4cb54cb\") " pod="openshift-dns/dns-default-ptkj7"
Sep 11 12:15:02.557067 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.557043    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25006c65-374d-49ed-b247-a8e0e9833b34-cert\") pod \"ingress-canary-n5nml\" (UID: \"25006c65-374d-49ed-b247-a8e0e9833b34\") " pod="openshift-ingress-canary/ingress-canary-n5nml"
Sep 11 12:15:02.572366 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.572346    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55hrp\" (UniqueName: \"kubernetes.io/projected/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-kube-api-access-55hrp\") pod \"dns-default-ptkj7\" (UID: \"4e3ebd74-ae34-4842-8ba8-184dc4cb54cb\") " pod="openshift-dns/dns-default-ptkj7"
Sep 11 12:15:02.572666 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.572646    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5jsn\" (UniqueName: \"kubernetes.io/projected/25006c65-374d-49ed-b247-a8e0e9833b34-kube-api-access-l5jsn\") pod \"ingress-canary-n5nml\" (UID: \"25006c65-374d-49ed-b247-a8e0e9833b34\") " pod="openshift-ingress-canary/ingress-canary-n5nml"
Sep 11 12:15:02.605552 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.605528    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n5nml"
Sep 11 12:15:02.652911 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.652887    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ca688b-efaa-42de-b332-78df0daf026b-metrics-certs\") pod \"network-metrics-daemon-bdr64\" (UID: \"26ca688b-efaa-42de-b332-78df0daf026b\") " pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:15:02.653022 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.652942    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2r6g\" (UniqueName: \"kubernetes.io/projected/719c0ee1-cbea-489a-9591-dcfb3e68b4e1-kube-api-access-r2r6g\") pod \"network-check-target-kjxhz\" (UID: \"719c0ee1-cbea-489a-9591-dcfb3e68b4e1\") " pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:15:02.653022 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:02.652974    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b254d6a2-6adc-468d-b21c-fb5ca9232417-original-pull-secret\") pod \"global-pull-secret-syncer-6qbjc\" (UID: \"b254d6a2-6adc-468d-b21c-fb5ca9232417\") " pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:15:02.653128 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:02.653037    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:15:02.653128 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:02.653071    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 12:15:02.653128 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:02.653093    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 12:15:02.653128 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:02.653109    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 12:15:02.653128 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:02.653112    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ca688b-efaa-42de-b332-78df0daf026b-metrics-certs podName:26ca688b-efaa-42de-b332-78df0daf026b nodeName:}" failed. No retries permitted until 2026-09-11 12:15:34.653093552 +0000 UTC m=+66.215873299 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26ca688b-efaa-42de-b332-78df0daf026b-metrics-certs") pod "network-metrics-daemon-bdr64" (UID: "26ca688b-efaa-42de-b332-78df0daf026b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 12:15:02.653128 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:02.653121    2561 projected.go:196] Error preparing data for projected volume kube-api-access-r2r6g for pod openshift-network-diagnostics/network-check-target-kjxhz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:15:02.653128 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:02.653131    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b254d6a2-6adc-468d-b21c-fb5ca9232417-original-pull-secret podName:b254d6a2-6adc-468d-b21c-fb5ca9232417 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:34.653121884 +0000 UTC m=+66.215901631 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b254d6a2-6adc-468d-b21c-fb5ca9232417-original-pull-secret") pod "global-pull-secret-syncer-6qbjc" (UID: "b254d6a2-6adc-468d-b21c-fb5ca9232417") : object "kube-system"/"original-pull-secret" not registered
Sep 11 12:15:02.653480 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:02.653153    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/719c0ee1-cbea-489a-9591-dcfb3e68b4e1-kube-api-access-r2r6g podName:719c0ee1-cbea-489a-9591-dcfb3e68b4e1 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:34.653142762 +0000 UTC m=+66.215922507 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-r2r6g" (UniqueName: "kubernetes.io/projected/719c0ee1-cbea-489a-9591-dcfb3e68b4e1-kube-api-access-r2r6g") pod "network-check-target-kjxhz" (UID: "719c0ee1-cbea-489a-9591-dcfb3e68b4e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 12:15:03.057113 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:03.056930    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-metrics-tls\") pod \"dns-default-ptkj7\" (UID: \"4e3ebd74-ae34-4842-8ba8-184dc4cb54cb\") " pod="openshift-dns/dns-default-ptkj7"
Sep 11 12:15:03.057113 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:03.057099    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 12:15:03.057545 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:03.057178    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-metrics-tls podName:4e3ebd74-ae34-4842-8ba8-184dc4cb54cb nodeName:}" failed. No retries permitted until 2026-09-11 12:15:04.057156138 +0000 UTC m=+35.619935884 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-metrics-tls") pod "dns-default-ptkj7" (UID: "4e3ebd74-ae34-4842-8ba8-184dc4cb54cb") : secret "dns-default-metrics-tls" not found
Sep 11 12:15:03.781990 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:03.779938    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n5nml"]
Sep 11 12:15:03.786971 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:15:03.786946    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25006c65_374d_49ed_b247_a8e0e9833b34.slice/crio-c3e26cbf799d653c30833aaf86e944603fb84c565940d9eee77df3218951ad46 WatchSource:0}: Error finding container c3e26cbf799d653c30833aaf86e944603fb84c565940d9eee77df3218951ad46: Status 404 returned error can't find the container with id c3e26cbf799d653c30833aaf86e944603fb84c565940d9eee77df3218951ad46
Sep 11 12:15:03.926395 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:03.926369    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:15:03.926498 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:03.926382    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:15:03.926498 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:03.926478    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:15:03.929220 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:03.929199    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 12:15:03.929327 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:03.929231    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dhjsw\""
Sep 11 12:15:03.929327 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:03.929249    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 12:15:03.930266 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:03.930249    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 12:15:03.930486 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:03.930347    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jvbml\""
Sep 11 12:15:03.930486 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:03.930386    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 12:15:04.063913 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:04.063884    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-metrics-tls\") pod \"dns-default-ptkj7\" (UID: \"4e3ebd74-ae34-4842-8ba8-184dc4cb54cb\") " pod="openshift-dns/dns-default-ptkj7"
Sep 11 12:15:04.064360 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:04.064001    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 12:15:04.064360 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:04.064052    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-metrics-tls podName:4e3ebd74-ae34-4842-8ba8-184dc4cb54cb nodeName:}" failed. No retries permitted until 2026-09-11 12:15:06.064038881 +0000 UTC m=+37.626818627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-metrics-tls") pod "dns-default-ptkj7" (UID: "4e3ebd74-ae34-4842-8ba8-184dc4cb54cb") : secret "dns-default-metrics-tls" not found
Sep 11 12:15:04.116056 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:04.116030    2561 generic.go:356] "Generic (PLEG): container finished" podID="0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f" containerID="d9a0fcb1903e972b5951bb0e649a2748020e438fda440509dd6b9cec2ca4f5cc" exitCode=0
Sep 11 12:15:04.116147 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:04.116112    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vw24w" event={"ID":"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f","Type":"ContainerDied","Data":"d9a0fcb1903e972b5951bb0e649a2748020e438fda440509dd6b9cec2ca4f5cc"}
Sep 11 12:15:04.117309 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:04.117285    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n5nml" event={"ID":"25006c65-374d-49ed-b247-a8e0e9833b34","Type":"ContainerStarted","Data":"c3e26cbf799d653c30833aaf86e944603fb84c565940d9eee77df3218951ad46"}
Sep 11 12:15:05.123889 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:05.123853    2561 generic.go:356] "Generic (PLEG): container finished" podID="0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f" containerID="fd7e12f41392bd8e86d299278b377a4d5940c10f1a38fd42bc79ffe29ffd82ef" exitCode=0
Sep 11 12:15:05.124399 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:05.123901    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vw24w" event={"ID":"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f","Type":"ContainerDied","Data":"fd7e12f41392bd8e86d299278b377a4d5940c10f1a38fd42bc79ffe29ffd82ef"}
Sep 11 12:15:06.078064 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:06.078032    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-metrics-tls\") pod \"dns-default-ptkj7\" (UID: \"4e3ebd74-ae34-4842-8ba8-184dc4cb54cb\") " pod="openshift-dns/dns-default-ptkj7"
Sep 11 12:15:06.078269 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:06.078166    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 12:15:06.078269 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:06.078249    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-metrics-tls podName:4e3ebd74-ae34-4842-8ba8-184dc4cb54cb nodeName:}" failed. No retries permitted until 2026-09-11 12:15:10.078231895 +0000 UTC m=+41.641011652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-metrics-tls") pod "dns-default-ptkj7" (UID: "4e3ebd74-ae34-4842-8ba8-184dc4cb54cb") : secret "dns-default-metrics-tls" not found
Sep 11 12:15:06.130164 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:06.130129    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vw24w" event={"ID":"0a06bf44-c6a4-45cf-a7cb-deefbbf69d2f","Type":"ContainerStarted","Data":"83af5808728c56379e63fdf2fcd1fc8bf1e3ea1c77d2fc59113525dfef60124c"}
Sep 11 12:15:06.158859 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:06.158809    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vw24w" podStartSLOduration=8.605969443 podStartE2EDuration="37.15879313s" totalImagesPullingTime="28.552823687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:06.157857088 +0000 UTC m=+37.720636873" watchObservedRunningTime="2026-09-11 12:15:06.15879313 +0000 UTC m=+37.721572893"
Sep 11 12:15:07.659595 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:07.659428    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zhfbt"]
Sep 11 12:15:07.681199 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:07.681167    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zhfbt"]
Sep 11 12:15:07.681335 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:07.681292    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:07.683971 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:07.683941    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 12:15:07.683971 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:07.683951    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 12:15:07.684149 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:07.683998    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 12:15:07.684149 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:07.684021    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 12:15:07.684968 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:07.684951    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l5m57\""
Sep 11 12:15:07.788043 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:07.788016    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhfbt\" (UID: \"f4ef5ce8-4e5c-4e66-a862-af9d4159263f\") " pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:07.788043 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:07.788046    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxvg6\" (UniqueName: \"kubernetes.io/projected/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-kube-api-access-jxvg6\") pod \"insights-runtime-extractor-zhfbt\" (UID: \"f4ef5ce8-4e5c-4e66-a862-af9d4159263f\") " pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:07.788256 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:07.788076    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-crio-socket\") pod \"insights-runtime-extractor-zhfbt\" (UID: \"f4ef5ce8-4e5c-4e66-a862-af9d4159263f\") " pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:07.788256 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:07.788152    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-data-volume\") pod \"insights-runtime-extractor-zhfbt\" (UID: \"f4ef5ce8-4e5c-4e66-a862-af9d4159263f\") " pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:07.788256 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:07.788173    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhfbt\" (UID: \"f4ef5ce8-4e5c-4e66-a862-af9d4159263f\") " pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:07.889458 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:07.889429    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhfbt\" (UID: \"f4ef5ce8-4e5c-4e66-a862-af9d4159263f\") " pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:07.889458 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:07.889457    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxvg6\" (UniqueName: \"kubernetes.io/projected/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-kube-api-access-jxvg6\") pod \"insights-runtime-extractor-zhfbt\" (UID: \"f4ef5ce8-4e5c-4e66-a862-af9d4159263f\") " pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:07.889627 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:07.889486    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-crio-socket\") pod \"insights-runtime-extractor-zhfbt\" (UID: \"f4ef5ce8-4e5c-4e66-a862-af9d4159263f\") " pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:07.889627 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:07.889519    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-data-volume\") pod \"insights-runtime-extractor-zhfbt\" (UID: \"f4ef5ce8-4e5c-4e66-a862-af9d4159263f\") " pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:07.889627 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:07.889543    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhfbt\" (UID: \"f4ef5ce8-4e5c-4e66-a862-af9d4159263f\") " pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:07.889627 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:07.889544    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 12:15:07.889627 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:07.889618    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-kube-rbac-proxy-cm podName:f4ef5ce8-4e5c-4e66-a862-af9d4159263f nodeName:}" failed. No retries permitted until 2026-09-11 12:15:08.389601071 +0000 UTC m=+39.952380830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zhfbt" (UID: "f4ef5ce8-4e5c-4e66-a862-af9d4159263f") : configmap "kube-rbac-proxy" not found
Sep 11 12:15:07.889845 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:07.889671    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 12:15:07.889845 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:07.889714    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-crio-socket\") pod \"insights-runtime-extractor-zhfbt\" (UID: \"f4ef5ce8-4e5c-4e66-a862-af9d4159263f\") " pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:07.889845 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:07.889737    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-insights-runtime-extractor-tls podName:f4ef5ce8-4e5c-4e66-a862-af9d4159263f nodeName:}" failed. No retries permitted until 2026-09-11 12:15:08.389718369 +0000 UTC m=+39.952498124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhfbt" (UID: "f4ef5ce8-4e5c-4e66-a862-af9d4159263f") : secret "insights-runtime-extractor-tls" not found
Sep 11 12:15:07.889845 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:07.889826    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-data-volume\") pod \"insights-runtime-extractor-zhfbt\" (UID: \"f4ef5ce8-4e5c-4e66-a862-af9d4159263f\") " pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:07.903910 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:07.903887    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxvg6\" (UniqueName: \"kubernetes.io/projected/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-kube-api-access-jxvg6\") pod \"insights-runtime-extractor-zhfbt\" (UID: \"f4ef5ce8-4e5c-4e66-a862-af9d4159263f\") " pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:08.138438 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:08.138410    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n5nml" event={"ID":"25006c65-374d-49ed-b247-a8e0e9833b34","Type":"ContainerStarted","Data":"0e59f6df557b83ccad861de5f30243dbd3fd71ffba4f1689c257449f00bfd0c3"}
Sep 11 12:15:08.155262 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:08.155222    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-n5nml" podStartSLOduration=2.929579117 podStartE2EDuration="6.155208306s" totalImagesPullingTime="3.225629189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:15:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:08.154860063 +0000 UTC m=+39.717639826" watchObservedRunningTime="2026-09-11 12:15:08.155208306 +0000 UTC m=+39.717988063"
Sep 11 12:15:08.393337 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:08.393241    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhfbt\" (UID: \"f4ef5ce8-4e5c-4e66-a862-af9d4159263f\") " pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:08.393492 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:08.393390    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 12:15:08.393492 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:08.393414    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhfbt\" (UID: \"f4ef5ce8-4e5c-4e66-a862-af9d4159263f\") " pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:08.393492 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:08.393449    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-insights-runtime-extractor-tls podName:f4ef5ce8-4e5c-4e66-a862-af9d4159263f nodeName:}" failed. No retries permitted until 2026-09-11 12:15:09.393433444 +0000 UTC m=+40.956213185 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhfbt" (UID: "f4ef5ce8-4e5c-4e66-a862-af9d4159263f") : secret "insights-runtime-extractor-tls" not found
Sep 11 12:15:08.393492 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:08.393475    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 12:15:08.393641 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:08.393515    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-kube-rbac-proxy-cm podName:f4ef5ce8-4e5c-4e66-a862-af9d4159263f nodeName:}" failed. No retries permitted until 2026-09-11 12:15:09.393504297 +0000 UTC m=+40.956284043 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zhfbt" (UID: "f4ef5ce8-4e5c-4e66-a862-af9d4159263f") : configmap "kube-rbac-proxy" not found
Sep 11 12:15:08.908297 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:08.908266    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-8mz42"]
Sep 11 12:15:08.924052 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:08.924029    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-8mz42"]
Sep 11 12:15:08.924224 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:08.924136    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-8mz42"
Sep 11 12:15:08.927925 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:08.927902    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 11 12:15:08.928242 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:08.928222    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bcl5x\""
Sep 11 12:15:08.928929 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:08.928913    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 11 12:15:09.097767 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:09.097732    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj2s5\" (UniqueName: \"kubernetes.io/projected/8d390870-ad6f-41e8-b467-f09aecf5adb7-kube-api-access-qj2s5\") pod \"migrator-8b4d697c5-8mz42\" (UID: \"8d390870-ad6f-41e8-b467-f09aecf5adb7\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-8mz42"
Sep 11 12:15:09.198263 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:09.198176    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj2s5\" (UniqueName: \"kubernetes.io/projected/8d390870-ad6f-41e8-b467-f09aecf5adb7-kube-api-access-qj2s5\") pod \"migrator-8b4d697c5-8mz42\" (UID: \"8d390870-ad6f-41e8-b467-f09aecf5adb7\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-8mz42"
Sep 11 12:15:09.206582 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:09.206561    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj2s5\" (UniqueName: \"kubernetes.io/projected/8d390870-ad6f-41e8-b467-f09aecf5adb7-kube-api-access-qj2s5\") pod \"migrator-8b4d697c5-8mz42\" (UID: \"8d390870-ad6f-41e8-b467-f09aecf5adb7\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-8mz42"
Sep 11 12:15:09.234485 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:09.234462    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-8mz42"
Sep 11 12:15:09.399214 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:09.399168    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhfbt\" (UID: \"f4ef5ce8-4e5c-4e66-a862-af9d4159263f\") " pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:09.399383 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:09.399246    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhfbt\" (UID: \"f4ef5ce8-4e5c-4e66-a862-af9d4159263f\") " pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:09.399383 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:09.399310    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 12:15:09.399383 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:09.399332    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 12:15:09.399383 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:09.399361    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-kube-rbac-proxy-cm podName:f4ef5ce8-4e5c-4e66-a862-af9d4159263f nodeName:}" failed. No retries permitted until 2026-09-11 12:15:11.399347673 +0000 UTC m=+42.962127415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zhfbt" (UID: "f4ef5ce8-4e5c-4e66-a862-af9d4159263f") : configmap "kube-rbac-proxy" not found
Sep 11 12:15:09.399383 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:09.399385    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-insights-runtime-extractor-tls podName:f4ef5ce8-4e5c-4e66-a862-af9d4159263f nodeName:}" failed. No retries permitted until 2026-09-11 12:15:11.399371965 +0000 UTC m=+42.962151706 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhfbt" (UID: "f4ef5ce8-4e5c-4e66-a862-af9d4159263f") : secret "insights-runtime-extractor-tls" not found
Sep 11 12:15:09.520013 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:09.519981    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-8mz42"]
Sep 11 12:15:09.523046 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:15:09.523021    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d390870_ad6f_41e8_b467_f09aecf5adb7.slice/crio-49525df8aebcaa654d41b771d082863b76821b8663a9ef504e19fb07947dad99 WatchSource:0}: Error finding container 49525df8aebcaa654d41b771d082863b76821b8663a9ef504e19fb07947dad99: Status 404 returned error can't find the container with id 49525df8aebcaa654d41b771d082863b76821b8663a9ef504e19fb07947dad99
Sep 11 12:15:09.624111 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:09.624090    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cj75q_b5df1e85-4605-4418-a2cb-b88a26b5c5b3/dns-node-resolver/0.log"
Sep 11 12:15:10.104495 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:10.104461    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-metrics-tls\") pod \"dns-default-ptkj7\" (UID: \"4e3ebd74-ae34-4842-8ba8-184dc4cb54cb\") " pod="openshift-dns/dns-default-ptkj7"
Sep 11 12:15:10.104866 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:10.104615    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 12:15:10.104866 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:10.104680    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-metrics-tls podName:4e3ebd74-ae34-4842-8ba8-184dc4cb54cb nodeName:}" failed. No retries permitted until 2026-09-11 12:15:18.104663783 +0000 UTC m=+49.667443524 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-metrics-tls") pod "dns-default-ptkj7" (UID: "4e3ebd74-ae34-4842-8ba8-184dc4cb54cb") : secret "dns-default-metrics-tls" not found
Sep 11 12:15:10.144143 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:10.144114    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-8mz42" event={"ID":"8d390870-ad6f-41e8-b467-f09aecf5adb7","Type":"ContainerStarted","Data":"49525df8aebcaa654d41b771d082863b76821b8663a9ef504e19fb07947dad99"}
Sep 11 12:15:10.424505 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:10.424437    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5bxb8_455e42df-85a6-4c25-a449-e12375d132e6/node-ca/0.log"
Sep 11 12:15:11.226091 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:11.226059    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n5nml_25006c65-374d-49ed-b247-a8e0e9833b34/serve-healthcheck-canary/0.log"
Sep 11 12:15:11.413668 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:11.413619    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhfbt\" (UID: \"f4ef5ce8-4e5c-4e66-a862-af9d4159263f\") " pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:11.413831 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:11.413701    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhfbt\" (UID: \"f4ef5ce8-4e5c-4e66-a862-af9d4159263f\") " pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:11.413831 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:11.413744    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 12:15:11.413831 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:11.413803    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-kube-rbac-proxy-cm podName:f4ef5ce8-4e5c-4e66-a862-af9d4159263f nodeName:}" failed. No retries permitted until 2026-09-11 12:15:15.413789019 +0000 UTC m=+46.976568764 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zhfbt" (UID: "f4ef5ce8-4e5c-4e66-a862-af9d4159263f") : configmap "kube-rbac-proxy" not found
Sep 11 12:15:11.413831 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:11.413804    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 12:15:11.413831 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:11.413831    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-insights-runtime-extractor-tls podName:f4ef5ce8-4e5c-4e66-a862-af9d4159263f nodeName:}" failed. No retries permitted until 2026-09-11 12:15:15.413824796 +0000 UTC m=+46.976604536 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhfbt" (UID: "f4ef5ce8-4e5c-4e66-a862-af9d4159263f") : secret "insights-runtime-extractor-tls" not found
Sep 11 12:15:13.152763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:13.152731    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-8mz42" event={"ID":"8d390870-ad6f-41e8-b467-f09aecf5adb7","Type":"ContainerStarted","Data":"a089285134e6bb1fb4392809a1f498ca9fdfce0ee97dc5b712369c370b745237"}
Sep 11 12:15:13.153155 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:13.152780    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-8mz42" event={"ID":"8d390870-ad6f-41e8-b467-f09aecf5adb7","Type":"ContainerStarted","Data":"56e1033fead13ddadcdf5225b6e9dcb4a0afac5d829bc228ae78f1cb0563c505"}
Sep 11 12:15:13.172056 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:13.172013    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-8mz42" podStartSLOduration=2.083900444 podStartE2EDuration="5.171998685s" totalImagesPullingTime="3.088098241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:13.171295509 +0000 UTC m=+44.734075272" watchObservedRunningTime="2026-09-11 12:15:13.171998685 +0000 UTC m=+44.734778448"
Sep 11 12:15:15.442882 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:15.442842    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhfbt\" (UID: \"f4ef5ce8-4e5c-4e66-a862-af9d4159263f\") " pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:15.443258 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:15.442923    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhfbt\" (UID: \"f4ef5ce8-4e5c-4e66-a862-af9d4159263f\") " pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:15.443258 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:15.443002    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 12:15:15.443258 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:15.443079    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-insights-runtime-extractor-tls podName:f4ef5ce8-4e5c-4e66-a862-af9d4159263f nodeName:}" failed. No retries permitted until 2026-09-11 12:15:23.443059705 +0000 UTC m=+55.005839464 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zhfbt" (UID: "f4ef5ce8-4e5c-4e66-a862-af9d4159263f") : secret "insights-runtime-extractor-tls" not found
Sep 11 12:15:15.443557 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:15.443537    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zhfbt\" (UID: \"f4ef5ce8-4e5c-4e66-a862-af9d4159263f\") " pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:18.163980 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:18.163947    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-metrics-tls\") pod \"dns-default-ptkj7\" (UID: \"4e3ebd74-ae34-4842-8ba8-184dc4cb54cb\") " pod="openshift-dns/dns-default-ptkj7"
Sep 11 12:15:18.164538 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:18.164087    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 12:15:18.164538 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:18.164150    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-metrics-tls podName:4e3ebd74-ae34-4842-8ba8-184dc4cb54cb nodeName:}" failed. No retries permitted until 2026-09-11 12:15:34.16413341 +0000 UTC m=+65.726913170 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-metrics-tls") pod "dns-default-ptkj7" (UID: "4e3ebd74-ae34-4842-8ba8-184dc4cb54cb") : secret "dns-default-metrics-tls" not found
Sep 11 12:15:23.501607 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:23.501567    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhfbt\" (UID: \"f4ef5ce8-4e5c-4e66-a862-af9d4159263f\") " pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:23.503820 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:23.503799    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4ef5ce8-4e5c-4e66-a862-af9d4159263f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zhfbt\" (UID: \"f4ef5ce8-4e5c-4e66-a862-af9d4159263f\") " pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:23.590263 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:23.590238    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zhfbt"
Sep 11 12:15:23.859507 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:23.859401    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zhfbt"]
Sep 11 12:15:23.862343 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:15:23.862315    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ef5ce8_4e5c_4e66_a862_af9d4159263f.slice/crio-040ca00f06b84aeacecd06ded328b3acfa171cde6b29c6796055d732fbff7997 WatchSource:0}: Error finding container 040ca00f06b84aeacecd06ded328b3acfa171cde6b29c6796055d732fbff7997: Status 404 returned error can't find the container with id 040ca00f06b84aeacecd06ded328b3acfa171cde6b29c6796055d732fbff7997
Sep 11 12:15:24.180857 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:24.180756    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zhfbt" event={"ID":"f4ef5ce8-4e5c-4e66-a862-af9d4159263f","Type":"ContainerStarted","Data":"d64b624cec4e8fdbcf9c1cd87c1b0f16b74b7f6af2c470d1dec918dd9a6fc958"}
Sep 11 12:15:24.180857 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:24.180839    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zhfbt" event={"ID":"f4ef5ce8-4e5c-4e66-a862-af9d4159263f","Type":"ContainerStarted","Data":"040ca00f06b84aeacecd06ded328b3acfa171cde6b29c6796055d732fbff7997"}
Sep 11 12:15:25.184666 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:25.184588    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zhfbt" event={"ID":"f4ef5ce8-4e5c-4e66-a862-af9d4159263f","Type":"ContainerStarted","Data":"9b2fbbcc04a8ec2549152af8df4e54e079bde7be99747f7d808c1c9a0dd29200"}
Sep 11 12:15:27.195166 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:27.193643    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zhfbt" event={"ID":"f4ef5ce8-4e5c-4e66-a862-af9d4159263f","Type":"ContainerStarted","Data":"bc09fda52c82050e8df63c401f5ba2687351324b5e952e0a6e0f9080ed56dada"}
Sep 11 12:15:27.218238 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:27.218167    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zhfbt" podStartSLOduration=17.856228982 podStartE2EDuration="20.218154461s" totalImagesPullingTime="2.361925479s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:15:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:27.218067994 +0000 UTC m=+58.780847757" watchObservedRunningTime="2026-09-11 12:15:27.218154461 +0000 UTC m=+58.780934217"
Sep 11 12:15:28.106592 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:28.106564    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bcfm6"
Sep 11 12:15:30.261707 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.261673    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-n9nrp"]
Sep 11 12:15:30.264140 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.264124    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-n9nrp"
Sep 11 12:15:30.274097 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.274045    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 11 12:15:30.274097 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.274058    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-b88wx\""
Sep 11 12:15:30.274290 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.274270    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 11 12:15:30.275573 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.275554    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64578868cb-v6c4b"]
Sep 11 12:15:30.277535 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.277516    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp"]
Sep 11 12:15:30.277641 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.277626    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64578868cb-v6c4b"
Sep 11 12:15:30.279470 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.279448    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp"
Sep 11 12:15:30.289132 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.289113    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 12:15:30.289252 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.289141    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 12:15:30.289328 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.289169    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 12:15:30.289328 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.289316    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-8mn29\""
Sep 11 12:15:30.289884 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.289862    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 11 12:15:30.289979 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.289869    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 11 12:15:30.290082 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.290060    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-n9nrp"]
Sep 11 12:15:30.297137 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.297117    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64578868cb-v6c4b"]
Sep 11 12:15:30.306392 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.306370    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp"]
Sep 11 12:15:30.448771 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.448741    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a11c300d-0b9e-4b28-89dc-72ce3c1254b5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-64578868cb-v6c4b\" (UID: \"a11c300d-0b9e-4b28-89dc-72ce3c1254b5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64578868cb-v6c4b"
Sep 11 12:15:30.448771 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.448775    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/188f0ec4-970e-4d5b-85ea-a4cdd0f3b106-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-n9nrp\" (UID: \"188f0ec4-970e-4d5b-85ea-a4cdd0f3b106\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-n9nrp"
Sep 11 12:15:30.448991 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.448794    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/188f0ec4-970e-4d5b-85ea-a4cdd0f3b106-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-n9nrp\" (UID: \"188f0ec4-970e-4d5b-85ea-a4cdd0f3b106\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-n9nrp"
Sep 11 12:15:30.448991 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.448812    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fdvn\" (UniqueName: \"kubernetes.io/projected/a11c300d-0b9e-4b28-89dc-72ce3c1254b5-kube-api-access-5fdvn\") pod \"managed-serviceaccount-addon-agent-64578868cb-v6c4b\" (UID: \"a11c300d-0b9e-4b28-89dc-72ce3c1254b5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64578868cb-v6c4b"
Sep 11 12:15:30.448991 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.448827    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3d0041c3-0e5e-48ae-98e1-2b9bf4c0ea67-klusterlet-config\") pod \"klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp\" (UID: \"3d0041c3-0e5e-48ae-98e1-2b9bf4c0ea67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp"
Sep 11 12:15:30.448991 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.448874    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xgr2\" (UniqueName: \"kubernetes.io/projected/3d0041c3-0e5e-48ae-98e1-2b9bf4c0ea67-kube-api-access-2xgr2\") pod \"klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp\" (UID: \"3d0041c3-0e5e-48ae-98e1-2b9bf4c0ea67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp"
Sep 11 12:15:30.448991 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.448971    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d0041c3-0e5e-48ae-98e1-2b9bf4c0ea67-tmp\") pod \"klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp\" (UID: \"3d0041c3-0e5e-48ae-98e1-2b9bf4c0ea67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp"
Sep 11 12:15:30.546380 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.546307    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6986b74764-wcmn5"]
Sep 11 12:15:30.549355 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.549318    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.549480 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.549448    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d0041c3-0e5e-48ae-98e1-2b9bf4c0ea67-tmp\") pod \"klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp\" (UID: \"3d0041c3-0e5e-48ae-98e1-2b9bf4c0ea67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp"
Sep 11 12:15:30.549546 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.549490    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a11c300d-0b9e-4b28-89dc-72ce3c1254b5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-64578868cb-v6c4b\" (UID: \"a11c300d-0b9e-4b28-89dc-72ce3c1254b5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64578868cb-v6c4b"
Sep 11 12:15:30.549546 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.549520    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/188f0ec4-970e-4d5b-85ea-a4cdd0f3b106-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-n9nrp\" (UID: \"188f0ec4-970e-4d5b-85ea-a4cdd0f3b106\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-n9nrp"
Sep 11 12:15:30.549650 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.549545    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/188f0ec4-970e-4d5b-85ea-a4cdd0f3b106-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-n9nrp\" (UID: \"188f0ec4-970e-4d5b-85ea-a4cdd0f3b106\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-n9nrp"
Sep 11 12:15:30.549650 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.549571    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fdvn\" (UniqueName: \"kubernetes.io/projected/a11c300d-0b9e-4b28-89dc-72ce3c1254b5-kube-api-access-5fdvn\") pod \"managed-serviceaccount-addon-agent-64578868cb-v6c4b\" (UID: \"a11c300d-0b9e-4b28-89dc-72ce3c1254b5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64578868cb-v6c4b"
Sep 11 12:15:30.549650 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.549593    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3d0041c3-0e5e-48ae-98e1-2b9bf4c0ea67-klusterlet-config\") pod \"klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp\" (UID: \"3d0041c3-0e5e-48ae-98e1-2b9bf4c0ea67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp"
Sep 11 12:15:30.549650 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.549625    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xgr2\" (UniqueName: \"kubernetes.io/projected/3d0041c3-0e5e-48ae-98e1-2b9bf4c0ea67-kube-api-access-2xgr2\") pod \"klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp\" (UID: \"3d0041c3-0e5e-48ae-98e1-2b9bf4c0ea67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp"
Sep 11 12:15:30.550231 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.550208    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d0041c3-0e5e-48ae-98e1-2b9bf4c0ea67-tmp\") pod \"klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp\" (UID: \"3d0041c3-0e5e-48ae-98e1-2b9bf4c0ea67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp"
Sep 11 12:15:30.550431 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.550408    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/188f0ec4-970e-4d5b-85ea-a4cdd0f3b106-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-n9nrp\" (UID: \"188f0ec4-970e-4d5b-85ea-a4cdd0f3b106\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-n9nrp"
Sep 11 12:15:30.552278 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.552260    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a11c300d-0b9e-4b28-89dc-72ce3c1254b5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-64578868cb-v6c4b\" (UID: \"a11c300d-0b9e-4b28-89dc-72ce3c1254b5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64578868cb-v6c4b"
Sep 11 12:15:30.552698 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.552676    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/188f0ec4-970e-4d5b-85ea-a4cdd0f3b106-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-n9nrp\" (UID: \"188f0ec4-970e-4d5b-85ea-a4cdd0f3b106\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-n9nrp"
Sep 11 12:15:30.552743 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.552725    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3d0041c3-0e5e-48ae-98e1-2b9bf4c0ea67-klusterlet-config\") pod \"klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp\" (UID: \"3d0041c3-0e5e-48ae-98e1-2b9bf4c0ea67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp"
Sep 11 12:15:30.564154 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.564138    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 12:15:30.564613 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.564601    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 12:15:30.566585 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.566572    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 12:15:30.569243 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.569227    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bmqgw\""
Sep 11 12:15:30.570447 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.570429    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6986b74764-wcmn5"]
Sep 11 12:15:30.572642 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.572622    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-n9nrp"
Sep 11 12:15:30.575329 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.575310    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 12:15:30.592569 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.592543    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xgr2\" (UniqueName: \"kubernetes.io/projected/3d0041c3-0e5e-48ae-98e1-2b9bf4c0ea67-kube-api-access-2xgr2\") pod \"klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp\" (UID: \"3d0041c3-0e5e-48ae-98e1-2b9bf4c0ea67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp"
Sep 11 12:15:30.593468 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.593446    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fdvn\" (UniqueName: \"kubernetes.io/projected/a11c300d-0b9e-4b28-89dc-72ce3c1254b5-kube-api-access-5fdvn\") pod \"managed-serviceaccount-addon-agent-64578868cb-v6c4b\" (UID: \"a11c300d-0b9e-4b28-89dc-72ce3c1254b5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64578868cb-v6c4b"
Sep 11 12:15:30.601511 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.601489    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64578868cb-v6c4b"
Sep 11 12:15:30.607506 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.607486    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp"
Sep 11 12:15:30.650546 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.650512    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21a8830b-a81f-488a-bb3e-8ad43abc7d50-bound-sa-token\") pod \"image-registry-6986b74764-wcmn5\" (UID: \"21a8830b-a81f-488a-bb3e-8ad43abc7d50\") " pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.650726 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.650560    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21a8830b-a81f-488a-bb3e-8ad43abc7d50-trusted-ca\") pod \"image-registry-6986b74764-wcmn5\" (UID: \"21a8830b-a81f-488a-bb3e-8ad43abc7d50\") " pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.650726 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.650603    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21a8830b-a81f-488a-bb3e-8ad43abc7d50-registry-certificates\") pod \"image-registry-6986b74764-wcmn5\" (UID: \"21a8830b-a81f-488a-bb3e-8ad43abc7d50\") " pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.650726 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.650636    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21a8830b-a81f-488a-bb3e-8ad43abc7d50-image-registry-private-configuration\") pod \"image-registry-6986b74764-wcmn5\" (UID: \"21a8830b-a81f-488a-bb3e-8ad43abc7d50\") " pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.650726 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.650665    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21a8830b-a81f-488a-bb3e-8ad43abc7d50-ca-trust-extracted\") pod \"image-registry-6986b74764-wcmn5\" (UID: \"21a8830b-a81f-488a-bb3e-8ad43abc7d50\") " pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.650726 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.650687    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21a8830b-a81f-488a-bb3e-8ad43abc7d50-installation-pull-secrets\") pod \"image-registry-6986b74764-wcmn5\" (UID: \"21a8830b-a81f-488a-bb3e-8ad43abc7d50\") " pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.650726 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.650718    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqh7t\" (UniqueName: \"kubernetes.io/projected/21a8830b-a81f-488a-bb3e-8ad43abc7d50-kube-api-access-cqh7t\") pod \"image-registry-6986b74764-wcmn5\" (UID: \"21a8830b-a81f-488a-bb3e-8ad43abc7d50\") " pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.651040 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.650763    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21a8830b-a81f-488a-bb3e-8ad43abc7d50-registry-tls\") pod \"image-registry-6986b74764-wcmn5\" (UID: \"21a8830b-a81f-488a-bb3e-8ad43abc7d50\") " pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.751858 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.751823    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21a8830b-a81f-488a-bb3e-8ad43abc7d50-image-registry-private-configuration\") pod \"image-registry-6986b74764-wcmn5\" (UID: \"21a8830b-a81f-488a-bb3e-8ad43abc7d50\") " pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.751858 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.751857    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21a8830b-a81f-488a-bb3e-8ad43abc7d50-ca-trust-extracted\") pod \"image-registry-6986b74764-wcmn5\" (UID: \"21a8830b-a81f-488a-bb3e-8ad43abc7d50\") " pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.752054 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.751875    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21a8830b-a81f-488a-bb3e-8ad43abc7d50-installation-pull-secrets\") pod \"image-registry-6986b74764-wcmn5\" (UID: \"21a8830b-a81f-488a-bb3e-8ad43abc7d50\") " pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.752054 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.751896    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqh7t\" (UniqueName: \"kubernetes.io/projected/21a8830b-a81f-488a-bb3e-8ad43abc7d50-kube-api-access-cqh7t\") pod \"image-registry-6986b74764-wcmn5\" (UID: \"21a8830b-a81f-488a-bb3e-8ad43abc7d50\") " pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.752054 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.751937    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21a8830b-a81f-488a-bb3e-8ad43abc7d50-registry-tls\") pod \"image-registry-6986b74764-wcmn5\" (UID: \"21a8830b-a81f-488a-bb3e-8ad43abc7d50\") " pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.752054 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.752014    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21a8830b-a81f-488a-bb3e-8ad43abc7d50-bound-sa-token\") pod \"image-registry-6986b74764-wcmn5\" (UID: \"21a8830b-a81f-488a-bb3e-8ad43abc7d50\") " pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.752054 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.752034    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21a8830b-a81f-488a-bb3e-8ad43abc7d50-trusted-ca\") pod \"image-registry-6986b74764-wcmn5\" (UID: \"21a8830b-a81f-488a-bb3e-8ad43abc7d50\") " pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.752310 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.752070    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21a8830b-a81f-488a-bb3e-8ad43abc7d50-registry-certificates\") pod \"image-registry-6986b74764-wcmn5\" (UID: \"21a8830b-a81f-488a-bb3e-8ad43abc7d50\") " pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.752365 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.752316    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21a8830b-a81f-488a-bb3e-8ad43abc7d50-ca-trust-extracted\") pod \"image-registry-6986b74764-wcmn5\" (UID: \"21a8830b-a81f-488a-bb3e-8ad43abc7d50\") " pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.753054 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.753028    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21a8830b-a81f-488a-bb3e-8ad43abc7d50-registry-certificates\") pod \"image-registry-6986b74764-wcmn5\" (UID: \"21a8830b-a81f-488a-bb3e-8ad43abc7d50\") " pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.753873 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.753850    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21a8830b-a81f-488a-bb3e-8ad43abc7d50-trusted-ca\") pod \"image-registry-6986b74764-wcmn5\" (UID: \"21a8830b-a81f-488a-bb3e-8ad43abc7d50\") " pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.754305 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.754286    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21a8830b-a81f-488a-bb3e-8ad43abc7d50-installation-pull-secrets\") pod \"image-registry-6986b74764-wcmn5\" (UID: \"21a8830b-a81f-488a-bb3e-8ad43abc7d50\") " pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.754385 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.754320    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21a8830b-a81f-488a-bb3e-8ad43abc7d50-image-registry-private-configuration\") pod \"image-registry-6986b74764-wcmn5\" (UID: \"21a8830b-a81f-488a-bb3e-8ad43abc7d50\") " pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.754648 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.754627    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21a8830b-a81f-488a-bb3e-8ad43abc7d50-registry-tls\") pod \"image-registry-6986b74764-wcmn5\" (UID: \"21a8830b-a81f-488a-bb3e-8ad43abc7d50\") " pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.765992 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.765974    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqh7t\" (UniqueName: \"kubernetes.io/projected/21a8830b-a81f-488a-bb3e-8ad43abc7d50-kube-api-access-cqh7t\") pod \"image-registry-6986b74764-wcmn5\" (UID: \"21a8830b-a81f-488a-bb3e-8ad43abc7d50\") " pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.767378 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.767358    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21a8830b-a81f-488a-bb3e-8ad43abc7d50-bound-sa-token\") pod \"image-registry-6986b74764-wcmn5\" (UID: \"21a8830b-a81f-488a-bb3e-8ad43abc7d50\") " pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.857903 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:15:30.857882    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188f0ec4_970e_4d5b_85ea_a4cdd0f3b106.slice/crio-2603dd39e95223017048fd54c8ab655d070ad550f3f3d9b4283a7e0cbffd5ea6 WatchSource:0}: Error finding container 2603dd39e95223017048fd54c8ab655d070ad550f3f3d9b4283a7e0cbffd5ea6: Status 404 returned error can't find the container with id 2603dd39e95223017048fd54c8ab655d070ad550f3f3d9b4283a7e0cbffd5ea6
Sep 11 12:15:30.864708 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.864687    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-n9nrp"]
Sep 11 12:15:30.865913 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.865885    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:30.906937 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.906801    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp"]
Sep 11 12:15:30.907524 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:30.907502    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64578868cb-v6c4b"]
Sep 11 12:15:30.908237 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:15:30.908214    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0041c3_0e5e_48ae_98e1_2b9bf4c0ea67.slice/crio-b97a3d31477d5f2f00b80d3dadf8f2f0f579058b0f54ce58d469b0b9c7716c3e WatchSource:0}: Error finding container b97a3d31477d5f2f00b80d3dadf8f2f0f579058b0f54ce58d469b0b9c7716c3e: Status 404 returned error can't find the container with id b97a3d31477d5f2f00b80d3dadf8f2f0f579058b0f54ce58d469b0b9c7716c3e
Sep 11 12:15:30.909394 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:15:30.909372    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda11c300d_0b9e_4b28_89dc_72ce3c1254b5.slice/crio-e63029250011d65a4463df7c0dd3c330032cdf2c4b347027648bdca1c6a7d10c WatchSource:0}: Error finding container e63029250011d65a4463df7c0dd3c330032cdf2c4b347027648bdca1c6a7d10c: Status 404 returned error can't find the container with id e63029250011d65a4463df7c0dd3c330032cdf2c4b347027648bdca1c6a7d10c
Sep 11 12:15:31.151739 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:31.151716    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6986b74764-wcmn5"]
Sep 11 12:15:31.153018 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:15:31.152995    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a8830b_a81f_488a_bb3e_8ad43abc7d50.slice/crio-65e4d63de0b221b733228b8f85ef4ee2b7a22392f3a210339bead11aa887afae WatchSource:0}: Error finding container 65e4d63de0b221b733228b8f85ef4ee2b7a22392f3a210339bead11aa887afae: Status 404 returned error can't find the container with id 65e4d63de0b221b733228b8f85ef4ee2b7a22392f3a210339bead11aa887afae
Sep 11 12:15:31.208964 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:31.208930    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-n9nrp" event={"ID":"188f0ec4-970e-4d5b-85ea-a4cdd0f3b106","Type":"ContainerStarted","Data":"2603dd39e95223017048fd54c8ab655d070ad550f3f3d9b4283a7e0cbffd5ea6"}
Sep 11 12:15:31.210231 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:31.210212    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp" event={"ID":"3d0041c3-0e5e-48ae-98e1-2b9bf4c0ea67","Type":"ContainerStarted","Data":"b97a3d31477d5f2f00b80d3dadf8f2f0f579058b0f54ce58d469b0b9c7716c3e"}
Sep 11 12:15:31.211230 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:31.211210    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64578868cb-v6c4b" event={"ID":"a11c300d-0b9e-4b28-89dc-72ce3c1254b5","Type":"ContainerStarted","Data":"e63029250011d65a4463df7c0dd3c330032cdf2c4b347027648bdca1c6a7d10c"}
Sep 11 12:15:31.212682 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:31.212663    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6986b74764-wcmn5" event={"ID":"21a8830b-a81f-488a-bb3e-8ad43abc7d50","Type":"ContainerStarted","Data":"6e7c01b946ca90a5b193576add0c6e75bbe2e039b9c8e6d668ba5b86aabad2cc"}
Sep 11 12:15:31.212762 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:31.212694    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6986b74764-wcmn5" event={"ID":"21a8830b-a81f-488a-bb3e-8ad43abc7d50","Type":"ContainerStarted","Data":"65e4d63de0b221b733228b8f85ef4ee2b7a22392f3a210339bead11aa887afae"}
Sep 11 12:15:31.212811 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:31.212805    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:31.246866 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:31.246814    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6986b74764-wcmn5" podStartSLOduration=1.24679921 podStartE2EDuration="1.24679921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:31.246201214 +0000 UTC m=+62.808980968" watchObservedRunningTime="2026-09-11 12:15:31.24679921 +0000 UTC m=+62.809578963"
Sep 11 12:15:33.221882 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:33.221839    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-n9nrp" event={"ID":"188f0ec4-970e-4d5b-85ea-a4cdd0f3b106","Type":"ContainerStarted","Data":"b9de4fb487f6aaec2012eb06b4890865e8ae3b9dd84971db4de1a0d8720830fd"}
Sep 11 12:15:33.252050 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:33.249053    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-n9nrp" podStartSLOduration=1.864349817 podStartE2EDuration="3.249031937s" totalImagesPullingTime="1.38468212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:33.248265363 +0000 UTC m=+64.811045127" watchObservedRunningTime="2026-09-11 12:15:33.249031937 +0000 UTC m=+64.811811701"
Sep 11 12:15:34.178760 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:34.178707    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-metrics-tls\") pod \"dns-default-ptkj7\" (UID: \"4e3ebd74-ae34-4842-8ba8-184dc4cb54cb\") " pod="openshift-dns/dns-default-ptkj7"
Sep 11 12:15:34.181361 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:34.181333    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e3ebd74-ae34-4842-8ba8-184dc4cb54cb-metrics-tls\") pod \"dns-default-ptkj7\" (UID: \"4e3ebd74-ae34-4842-8ba8-184dc4cb54cb\") " pod="openshift-dns/dns-default-ptkj7"
Sep 11 12:15:34.395795 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:34.395767    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lgchp\""
Sep 11 12:15:34.403348 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:34.403323    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ptkj7"
Sep 11 12:15:34.682913 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:34.682872    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2r6g\" (UniqueName: \"kubernetes.io/projected/719c0ee1-cbea-489a-9591-dcfb3e68b4e1-kube-api-access-r2r6g\") pod \"network-check-target-kjxhz\" (UID: \"719c0ee1-cbea-489a-9591-dcfb3e68b4e1\") " pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:15:34.683111 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:34.682936    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b254d6a2-6adc-468d-b21c-fb5ca9232417-original-pull-secret\") pod \"global-pull-secret-syncer-6qbjc\" (UID: \"b254d6a2-6adc-468d-b21c-fb5ca9232417\") " pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:15:34.683111 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:34.682988    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ca688b-efaa-42de-b332-78df0daf026b-metrics-certs\") pod \"network-metrics-daemon-bdr64\" (UID: \"26ca688b-efaa-42de-b332-78df0daf026b\") " pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:15:34.685980 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:34.685907    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 12:15:34.686099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:34.685979    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 12:15:34.686240 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:34.686216    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 12:15:34.696320 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:34.696236    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ca688b-efaa-42de-b332-78df0daf026b-metrics-certs\") pod \"network-metrics-daemon-bdr64\" (UID: \"26ca688b-efaa-42de-b332-78df0daf026b\") " pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:15:34.696320 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:34.696253    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b254d6a2-6adc-468d-b21c-fb5ca9232417-original-pull-secret\") pod \"global-pull-secret-syncer-6qbjc\" (UID: \"b254d6a2-6adc-468d-b21c-fb5ca9232417\") " pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:15:34.696502 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:34.696381    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 12:15:34.707207 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:34.707171    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2r6g\" (UniqueName: \"kubernetes.io/projected/719c0ee1-cbea-489a-9591-dcfb3e68b4e1-kube-api-access-r2r6g\") pod \"network-check-target-kjxhz\" (UID: \"719c0ee1-cbea-489a-9591-dcfb3e68b4e1\") " pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:15:34.839942 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:34.839912    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dhjsw\""
Sep 11 12:15:34.843149 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:34.843125    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qbjc"
Sep 11 12:15:34.847089 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:34.847067    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:15:34.849537 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:34.849520    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jvbml\""
Sep 11 12:15:34.857596 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:34.857571    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdr64"
Sep 11 12:15:36.036470 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:36.036445    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kjxhz"]
Sep 11 12:15:36.040230 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:15:36.039599    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719c0ee1_cbea_489a_9591_dcfb3e68b4e1.slice/crio-872710e16dc8c8339f0d23c44b87d2a22fc1ff15cf4f14138692605c38ebd06b WatchSource:0}: Error finding container 872710e16dc8c8339f0d23c44b87d2a22fc1ff15cf4f14138692605c38ebd06b: Status 404 returned error can't find the container with id 872710e16dc8c8339f0d23c44b87d2a22fc1ff15cf4f14138692605c38ebd06b
Sep 11 12:15:36.040230 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:36.040066    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bdr64"]
Sep 11 12:15:36.040919 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:36.040888    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ptkj7"]
Sep 11 12:15:36.042493 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:36.042394    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6qbjc"]
Sep 11 12:15:36.042493 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:15:36.042444    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ca688b_efaa_42de_b332_78df0daf026b.slice/crio-c00a4ef7a57947e1bdf56553f3a343e3a977cb62d41f5478579c209d520ab8e2 WatchSource:0}: Error finding container c00a4ef7a57947e1bdf56553f3a343e3a977cb62d41f5478579c209d520ab8e2: Status 404 returned error can't find the container with id c00a4ef7a57947e1bdf56553f3a343e3a977cb62d41f5478579c209d520ab8e2
Sep 11 12:15:36.043000 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:15:36.042974    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e3ebd74_ae34_4842_8ba8_184dc4cb54cb.slice/crio-d10f7ab109dd5e00d2993937ca1a5e1a71a773f5ff15feaed5b0a86b864e49ff WatchSource:0}: Error finding container d10f7ab109dd5e00d2993937ca1a5e1a71a773f5ff15feaed5b0a86b864e49ff: Status 404 returned error can't find the container with id d10f7ab109dd5e00d2993937ca1a5e1a71a773f5ff15feaed5b0a86b864e49ff
Sep 11 12:15:36.044766 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:15:36.044722    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb254d6a2_6adc_468d_b21c_fb5ca9232417.slice/crio-5f4d1d206e0b010907cc7da30fb4887d07e513fe4d074992ed7e0ef5f089589a WatchSource:0}: Error finding container 5f4d1d206e0b010907cc7da30fb4887d07e513fe4d074992ed7e0ef5f089589a: Status 404 returned error can't find the container with id 5f4d1d206e0b010907cc7da30fb4887d07e513fe4d074992ed7e0ef5f089589a
Sep 11 12:15:36.234012 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:36.233972    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp" event={"ID":"3d0041c3-0e5e-48ae-98e1-2b9bf4c0ea67","Type":"ContainerStarted","Data":"b6e563dee0212d195f8a50a4e24a2280635facbee68d2c00726365c6cd6d0e11"}
Sep 11 12:15:36.234205 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:36.234160    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp"
Sep 11 12:15:36.235399 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:36.235368    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kjxhz" event={"ID":"719c0ee1-cbea-489a-9591-dcfb3e68b4e1","Type":"ContainerStarted","Data":"872710e16dc8c8339f0d23c44b87d2a22fc1ff15cf4f14138692605c38ebd06b"}
Sep 11 12:15:36.236122 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:36.236107    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp"
Sep 11 12:15:36.236947 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:36.236913    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64578868cb-v6c4b" event={"ID":"a11c300d-0b9e-4b28-89dc-72ce3c1254b5","Type":"ContainerStarted","Data":"8fb1688656aced54d57cdfc3882ce60eb33faad3e5b16cea06064cd756afb99e"}
Sep 11 12:15:36.238051 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:36.238031    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ptkj7" event={"ID":"4e3ebd74-ae34-4842-8ba8-184dc4cb54cb","Type":"ContainerStarted","Data":"d10f7ab109dd5e00d2993937ca1a5e1a71a773f5ff15feaed5b0a86b864e49ff"}
Sep 11 12:15:36.239134 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:36.239114    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6qbjc" event={"ID":"b254d6a2-6adc-468d-b21c-fb5ca9232417","Type":"ContainerStarted","Data":"5f4d1d206e0b010907cc7da30fb4887d07e513fe4d074992ed7e0ef5f089589a"}
Sep 11 12:15:36.240069 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:36.240051    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bdr64" event={"ID":"26ca688b-efaa-42de-b332-78df0daf026b","Type":"ContainerStarted","Data":"c00a4ef7a57947e1bdf56553f3a343e3a977cb62d41f5478579c209d520ab8e2"}
Sep 11 12:15:36.253497 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:36.253456    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fd6f6d-rg7vp" podStartSLOduration=1.388093773 podStartE2EDuration="6.25344388s" totalImagesPullingTime="4.865350107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:36.25244616 +0000 UTC m=+67.815225946" watchObservedRunningTime="2026-09-11 12:15:36.25344388 +0000 UTC m=+67.816223669"
Sep 11 12:15:36.286431 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:36.286362    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64578868cb-v6c4b" podStartSLOduration=1.441711275 podStartE2EDuration="6.286349947s" totalImagesPullingTime="4.844638672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:36.286035588 +0000 UTC m=+67.848815351" watchObservedRunningTime="2026-09-11 12:15:36.286349947 +0000 UTC m=+67.849129704"
Sep 11 12:15:39.253526 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:39.253479    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bdr64" event={"ID":"26ca688b-efaa-42de-b332-78df0daf026b","Type":"ContainerStarted","Data":"593c90329b3dae10b0ebd91ffbd4dbed065210651f420639a899f3ea34fb6545"}
Sep 11 12:15:41.266251 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:41.266164    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kjxhz" event={"ID":"719c0ee1-cbea-489a-9591-dcfb3e68b4e1","Type":"ContainerStarted","Data":"6d2ee539bb84cae4525d0e0b6b61d2c68c20acd4a739b4f3b40929eec14e892e"}
Sep 11 12:15:41.266713 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:41.266476    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:15:41.268331 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:41.268301    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ptkj7" event={"ID":"4e3ebd74-ae34-4842-8ba8-184dc4cb54cb","Type":"ContainerStarted","Data":"62f7074069069b79f9089016af4f2e0ef1222953421ff127550df1a497b5b8a7"}
Sep 11 12:15:41.268451 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:41.268336    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ptkj7" event={"ID":"4e3ebd74-ae34-4842-8ba8-184dc4cb54cb","Type":"ContainerStarted","Data":"65ba5cc4a8b5943e4a5caee295dd2d63cbbb5477fac51fd474f7a6eccd4965e7"}
Sep 11 12:15:41.268451 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:41.268372    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ptkj7"
Sep 11 12:15:41.270514 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:41.270334    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6qbjc" event={"ID":"b254d6a2-6adc-468d-b21c-fb5ca9232417","Type":"ContainerStarted","Data":"ffca5b57e63e935b130c9bd61770f6488cdacf5468dbfdcb83594cffff7464be"}
Sep 11 12:15:41.271798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:41.271777    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bdr64" event={"ID":"26ca688b-efaa-42de-b332-78df0daf026b","Type":"ContainerStarted","Data":"c2e7ee4b21fdd150d6b072296134187825da1692a4f1f416e563af2eb50e638a"}
Sep 11 12:15:41.288275 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:41.288237    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kjxhz" podStartSLOduration=67.277622912 podStartE2EDuration="1m12.288224721s" totalImagesPullingTime="5.010601809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:41.287472942 +0000 UTC m=+72.850252706" watchObservedRunningTime="2026-09-11 12:15:41.288224721 +0000 UTC m=+72.851004481"
Sep 11 12:15:41.343698 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:41.343651    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ptkj7" podStartSLOduration=34.35239529 podStartE2EDuration="39.343637247s" totalImagesPullingTime="4.991241957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:15:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:41.342679189 +0000 UTC m=+72.905458952" watchObservedRunningTime="2026-09-11 12:15:41.343637247 +0000 UTC m=+72.906417024"
Sep 11 12:15:41.343865 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:41.343835    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bdr64" podStartSLOduration=70.070990879 podStartE2EDuration="1m12.343829071s" totalImagesPullingTime="2.272838192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:41.317804644 +0000 UTC m=+72.880584407" watchObservedRunningTime="2026-09-11 12:15:41.343829071 +0000 UTC m=+72.906608834"
Sep 11 12:15:41.361741 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:41.361698    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6qbjc" podStartSLOduration=67.354285618 podStartE2EDuration="1m12.361684532s" totalImagesPullingTime="5.007398914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:41.359927861 +0000 UTC m=+72.922707623" watchObservedRunningTime="2026-09-11 12:15:41.361684532 +0000 UTC m=+72.924464296"
Sep 11 12:15:45.250686 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.250651    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wxcr6"]
Sep 11 12:15:45.254222 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.254204    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.256727 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.256707    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 12:15:45.256845 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.256712    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 12:15:45.257522 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.257506    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bv4vc\""
Sep 11 12:15:45.257628 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.257548    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 12:15:45.257628 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.257560    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 12:15:45.257628 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.257605    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 12:15:45.257628 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.257568    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 12:15:45.360199 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.360168    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/653ad827-71d7-4cc1-bacd-ea44b590787c-node-exporter-wtmp\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.360381 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.360215    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t86jq\" (UniqueName: \"kubernetes.io/projected/653ad827-71d7-4cc1-bacd-ea44b590787c-kube-api-access-t86jq\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.360381 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.360233    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/653ad827-71d7-4cc1-bacd-ea44b590787c-root\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.360381 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.360253    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/653ad827-71d7-4cc1-bacd-ea44b590787c-node-exporter-tls\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.360381 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.360301    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/653ad827-71d7-4cc1-bacd-ea44b590787c-sys\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.360381 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.360348    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/653ad827-71d7-4cc1-bacd-ea44b590787c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.360558 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.360405    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/653ad827-71d7-4cc1-bacd-ea44b590787c-node-exporter-accelerators-collector-config\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.360558 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.360433    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/653ad827-71d7-4cc1-bacd-ea44b590787c-node-exporter-textfile\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.360558 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.360447    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/653ad827-71d7-4cc1-bacd-ea44b590787c-metrics-client-ca\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.461513 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.461477    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/653ad827-71d7-4cc1-bacd-ea44b590787c-node-exporter-wtmp\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.461513 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.461515    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t86jq\" (UniqueName: \"kubernetes.io/projected/653ad827-71d7-4cc1-bacd-ea44b590787c-kube-api-access-t86jq\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.461738 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.461532    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/653ad827-71d7-4cc1-bacd-ea44b590787c-root\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.461738 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.461552    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/653ad827-71d7-4cc1-bacd-ea44b590787c-node-exporter-tls\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.461738 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.461573    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/653ad827-71d7-4cc1-bacd-ea44b590787c-sys\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.461738 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.461589    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/653ad827-71d7-4cc1-bacd-ea44b590787c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.461738 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.461615    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/653ad827-71d7-4cc1-bacd-ea44b590787c-root\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.461738 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.461653    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/653ad827-71d7-4cc1-bacd-ea44b590787c-sys\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.461738 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.461654    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/653ad827-71d7-4cc1-bacd-ea44b590787c-node-exporter-wtmp\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.461738 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:45.461671    2561 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 12:15:45.461738 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.461680    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/653ad827-71d7-4cc1-bacd-ea44b590787c-node-exporter-accelerators-collector-config\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.461738 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.461722    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/653ad827-71d7-4cc1-bacd-ea44b590787c-node-exporter-textfile\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.461738 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:15:45.461747    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653ad827-71d7-4cc1-bacd-ea44b590787c-node-exporter-tls podName:653ad827-71d7-4cc1-bacd-ea44b590787c nodeName:}" failed. No retries permitted until 2026-09-11 12:15:45.961726246 +0000 UTC m=+77.524505997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/653ad827-71d7-4cc1-bacd-ea44b590787c-node-exporter-tls") pod "node-exporter-wxcr6" (UID: "653ad827-71d7-4cc1-bacd-ea44b590787c") : secret "node-exporter-tls" not found
Sep 11 12:15:45.462167 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.461773    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/653ad827-71d7-4cc1-bacd-ea44b590787c-metrics-client-ca\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.462167 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.462065    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/653ad827-71d7-4cc1-bacd-ea44b590787c-node-exporter-textfile\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.462322 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.462306    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/653ad827-71d7-4cc1-bacd-ea44b590787c-metrics-client-ca\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.462370 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.462354    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/653ad827-71d7-4cc1-bacd-ea44b590787c-node-exporter-accelerators-collector-config\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.463886 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.463867    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/653ad827-71d7-4cc1-bacd-ea44b590787c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.472414 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.472393    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t86jq\" (UniqueName: \"kubernetes.io/projected/653ad827-71d7-4cc1-bacd-ea44b590787c-kube-api-access-t86jq\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.964336 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.964300    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/653ad827-71d7-4cc1-bacd-ea44b590787c-node-exporter-tls\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:45.966672 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:45.966643    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/653ad827-71d7-4cc1-bacd-ea44b590787c-node-exporter-tls\") pod \"node-exporter-wxcr6\" (UID: \"653ad827-71d7-4cc1-bacd-ea44b590787c\") " pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:46.163711 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:46.163679    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wxcr6"
Sep 11 12:15:46.173487 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:15:46.173463    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653ad827_71d7_4cc1_bacd_ea44b590787c.slice/crio-fcdf66c0d2795ef4b092eeefa10e1210c46a30bafdeb486c62d593829ee59056 WatchSource:0}: Error finding container fcdf66c0d2795ef4b092eeefa10e1210c46a30bafdeb486c62d593829ee59056: Status 404 returned error can't find the container with id fcdf66c0d2795ef4b092eeefa10e1210c46a30bafdeb486c62d593829ee59056
Sep 11 12:15:46.292763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:46.292731    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wxcr6" event={"ID":"653ad827-71d7-4cc1-bacd-ea44b590787c","Type":"ContainerStarted","Data":"fcdf66c0d2795ef4b092eeefa10e1210c46a30bafdeb486c62d593829ee59056"}
Sep 11 12:15:47.291236 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.291206    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-59675fdc88-d92dw"]
Sep 11 12:15:47.294162 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.294146    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.297247 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.297221    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 11 12:15:47.297726 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.297705    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 11 12:15:47.297811 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.297726    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 11 12:15:47.297897 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.297876    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 11 12:15:47.297969 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.297954    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dd1mjbm29ecmh\""
Sep 11 12:15:47.298643 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.298625    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-h9mlt\""
Sep 11 12:15:47.298739 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.298704    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 11 12:15:47.306705 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.306687    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-59675fdc88-d92dw"]
Sep 11 12:15:47.374426 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.374402    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/32e2be1f-f47c-4158-bdb3-de6e93f42cf9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59675fdc88-d92dw\" (UID: \"32e2be1f-f47c-4158-bdb3-de6e93f42cf9\") " pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.374553 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.374433    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/32e2be1f-f47c-4158-bdb3-de6e93f42cf9-secret-thanos-querier-tls\") pod \"thanos-querier-59675fdc88-d92dw\" (UID: \"32e2be1f-f47c-4158-bdb3-de6e93f42cf9\") " pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.374553 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.374450    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/32e2be1f-f47c-4158-bdb3-de6e93f42cf9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59675fdc88-d92dw\" (UID: \"32e2be1f-f47c-4158-bdb3-de6e93f42cf9\") " pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.374553 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.374477    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32e2be1f-f47c-4158-bdb3-de6e93f42cf9-metrics-client-ca\") pod \"thanos-querier-59675fdc88-d92dw\" (UID: \"32e2be1f-f47c-4158-bdb3-de6e93f42cf9\") " pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.374673 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.374567    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrpfq\" (UniqueName: \"kubernetes.io/projected/32e2be1f-f47c-4158-bdb3-de6e93f42cf9-kube-api-access-jrpfq\") pod \"thanos-querier-59675fdc88-d92dw\" (UID: \"32e2be1f-f47c-4158-bdb3-de6e93f42cf9\") " pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.374673 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.374612    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/32e2be1f-f47c-4158-bdb3-de6e93f42cf9-secret-grpc-tls\") pod \"thanos-querier-59675fdc88-d92dw\" (UID: \"32e2be1f-f47c-4158-bdb3-de6e93f42cf9\") " pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.374673 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.374645    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32e2be1f-f47c-4158-bdb3-de6e93f42cf9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59675fdc88-d92dw\" (UID: \"32e2be1f-f47c-4158-bdb3-de6e93f42cf9\") " pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.374787 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.374724    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32e2be1f-f47c-4158-bdb3-de6e93f42cf9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59675fdc88-d92dw\" (UID: \"32e2be1f-f47c-4158-bdb3-de6e93f42cf9\") " pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.475717 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.475689    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32e2be1f-f47c-4158-bdb3-de6e93f42cf9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59675fdc88-d92dw\" (UID: \"32e2be1f-f47c-4158-bdb3-de6e93f42cf9\") " pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.475837 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.475722    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/32e2be1f-f47c-4158-bdb3-de6e93f42cf9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59675fdc88-d92dw\" (UID: \"32e2be1f-f47c-4158-bdb3-de6e93f42cf9\") " pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.475837 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.475745    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/32e2be1f-f47c-4158-bdb3-de6e93f42cf9-secret-thanos-querier-tls\") pod \"thanos-querier-59675fdc88-d92dw\" (UID: \"32e2be1f-f47c-4158-bdb3-de6e93f42cf9\") " pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.475837 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.475767    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/32e2be1f-f47c-4158-bdb3-de6e93f42cf9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59675fdc88-d92dw\" (UID: \"32e2be1f-f47c-4158-bdb3-de6e93f42cf9\") " pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.475837 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.475800    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32e2be1f-f47c-4158-bdb3-de6e93f42cf9-metrics-client-ca\") pod \"thanos-querier-59675fdc88-d92dw\" (UID: \"32e2be1f-f47c-4158-bdb3-de6e93f42cf9\") " pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.475837 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.475824    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrpfq\" (UniqueName: \"kubernetes.io/projected/32e2be1f-f47c-4158-bdb3-de6e93f42cf9-kube-api-access-jrpfq\") pod \"thanos-querier-59675fdc88-d92dw\" (UID: \"32e2be1f-f47c-4158-bdb3-de6e93f42cf9\") " pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.476113 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.475848    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/32e2be1f-f47c-4158-bdb3-de6e93f42cf9-secret-grpc-tls\") pod \"thanos-querier-59675fdc88-d92dw\" (UID: \"32e2be1f-f47c-4158-bdb3-de6e93f42cf9\") " pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.476113 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.475871    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32e2be1f-f47c-4158-bdb3-de6e93f42cf9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59675fdc88-d92dw\" (UID: \"32e2be1f-f47c-4158-bdb3-de6e93f42cf9\") " pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.476655 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.476629    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32e2be1f-f47c-4158-bdb3-de6e93f42cf9-metrics-client-ca\") pod \"thanos-querier-59675fdc88-d92dw\" (UID: \"32e2be1f-f47c-4158-bdb3-de6e93f42cf9\") " pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.478309 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.478241    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/32e2be1f-f47c-4158-bdb3-de6e93f42cf9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59675fdc88-d92dw\" (UID: \"32e2be1f-f47c-4158-bdb3-de6e93f42cf9\") " pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.478436 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.478415    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32e2be1f-f47c-4158-bdb3-de6e93f42cf9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59675fdc88-d92dw\" (UID: \"32e2be1f-f47c-4158-bdb3-de6e93f42cf9\") " pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.478607 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.478584    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32e2be1f-f47c-4158-bdb3-de6e93f42cf9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59675fdc88-d92dw\" (UID: \"32e2be1f-f47c-4158-bdb3-de6e93f42cf9\") " pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.478705 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.478683    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/32e2be1f-f47c-4158-bdb3-de6e93f42cf9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59675fdc88-d92dw\" (UID: \"32e2be1f-f47c-4158-bdb3-de6e93f42cf9\") " pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.478742 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.478722    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/32e2be1f-f47c-4158-bdb3-de6e93f42cf9-secret-thanos-querier-tls\") pod \"thanos-querier-59675fdc88-d92dw\" (UID: \"32e2be1f-f47c-4158-bdb3-de6e93f42cf9\") " pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.478839 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.478822    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/32e2be1f-f47c-4158-bdb3-de6e93f42cf9-secret-grpc-tls\") pod \"thanos-querier-59675fdc88-d92dw\" (UID: \"32e2be1f-f47c-4158-bdb3-de6e93f42cf9\") " pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.491065 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.491044    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrpfq\" (UniqueName: \"kubernetes.io/projected/32e2be1f-f47c-4158-bdb3-de6e93f42cf9-kube-api-access-jrpfq\") pod \"thanos-querier-59675fdc88-d92dw\" (UID: \"32e2be1f-f47c-4158-bdb3-de6e93f42cf9\") " pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.606835 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.606756    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:47.884440 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:47.884418    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-59675fdc88-d92dw"]
Sep 11 12:15:47.886453 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:15:47.886425    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e2be1f_f47c_4158_bdb3_de6e93f42cf9.slice/crio-3a58455d58b59aa50c4c09ca62b6ebdb7a5244c96f53e07fa6d0ced3cadf30ca WatchSource:0}: Error finding container 3a58455d58b59aa50c4c09ca62b6ebdb7a5244c96f53e07fa6d0ced3cadf30ca: Status 404 returned error can't find the container with id 3a58455d58b59aa50c4c09ca62b6ebdb7a5244c96f53e07fa6d0ced3cadf30ca
Sep 11 12:15:48.300200 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:48.300159    2561 generic.go:356] "Generic (PLEG): container finished" podID="653ad827-71d7-4cc1-bacd-ea44b590787c" containerID="96bb0dcf45814869176eaf8a140611ed3829da648d76103e3cd5d84236a3b3d3" exitCode=0
Sep 11 12:15:48.300605 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:48.300251    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wxcr6" event={"ID":"653ad827-71d7-4cc1-bacd-ea44b590787c","Type":"ContainerDied","Data":"96bb0dcf45814869176eaf8a140611ed3829da648d76103e3cd5d84236a3b3d3"}
Sep 11 12:15:48.301484 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:48.301465    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw" event={"ID":"32e2be1f-f47c-4158-bdb3-de6e93f42cf9","Type":"ContainerStarted","Data":"3a58455d58b59aa50c4c09ca62b6ebdb7a5244c96f53e07fa6d0ced3cadf30ca"}
Sep 11 12:15:49.306881 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.306842    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wxcr6" event={"ID":"653ad827-71d7-4cc1-bacd-ea44b590787c","Type":"ContainerStarted","Data":"65ad2ec2187ba73fb4ed74ab2781afe512e507c0f18e7616a3b87222780ce9d7"}
Sep 11 12:15:49.307306 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.306904    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wxcr6" event={"ID":"653ad827-71d7-4cc1-bacd-ea44b590787c","Type":"ContainerStarted","Data":"f3774178f4d0957ba2e2d14a638a322573ae017082069c16f5b7cbfaab0b48d1"}
Sep 11 12:15:49.341492 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.341443    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wxcr6" podStartSLOduration=3.195876035 podStartE2EDuration="4.341423698s" totalImagesPullingTime="1.08434692s" totalInitContainerRuntime="61.200743ms" isStatefulPod=true podCreationTimestamp="2026-09-11 12:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:49.340394162 +0000 UTC m=+80.903173926" watchObservedRunningTime="2026-09-11 12:15:49.341423698 +0000 UTC m=+80.904203495"
Sep 11 12:15:49.645835 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.645759    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"]
Sep 11 12:15:49.649054 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.649033    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:15:49.652156 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.652135    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 12:15:49.653081 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.653046    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 11 12:15:49.653081 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.653068    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 11 12:15:49.653252 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.653092    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8flt9\""
Sep 11 12:15:49.653252 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.653120    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-eompij84h1vi1\""
Sep 11 12:15:49.653430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.653414    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 11 12:15:49.661390 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.661365    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"]
Sep 11 12:15:49.696036 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.696008    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66mbd\" (UniqueName: \"kubernetes.io/projected/e8ac9128-16ec-4f9c-ac2c-bc3815231d52-kube-api-access-66mbd\") pod \"metrics-server-555dfb9dd8-9lkfd\" (UID: \"e8ac9128-16ec-4f9c-ac2c-bc3815231d52\") " pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:15:49.696168 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.696039    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e8ac9128-16ec-4f9c-ac2c-bc3815231d52-secret-metrics-server-tls\") pod \"metrics-server-555dfb9dd8-9lkfd\" (UID: \"e8ac9128-16ec-4f9c-ac2c-bc3815231d52\") " pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:15:49.696168 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.696065    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8ac9128-16ec-4f9c-ac2c-bc3815231d52-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-555dfb9dd8-9lkfd\" (UID: \"e8ac9128-16ec-4f9c-ac2c-bc3815231d52\") " pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:15:49.696168 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.696125    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e8ac9128-16ec-4f9c-ac2c-bc3815231d52-metrics-server-audit-profiles\") pod \"metrics-server-555dfb9dd8-9lkfd\" (UID: \"e8ac9128-16ec-4f9c-ac2c-bc3815231d52\") " pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:15:49.696168 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.696161    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e8ac9128-16ec-4f9c-ac2c-bc3815231d52-secret-metrics-server-client-certs\") pod \"metrics-server-555dfb9dd8-9lkfd\" (UID: \"e8ac9128-16ec-4f9c-ac2c-bc3815231d52\") " pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:15:49.696408 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.696204    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e8ac9128-16ec-4f9c-ac2c-bc3815231d52-audit-log\") pod \"metrics-server-555dfb9dd8-9lkfd\" (UID: \"e8ac9128-16ec-4f9c-ac2c-bc3815231d52\") " pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:15:49.696408 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.696241    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e8ac9128-16ec-4f9c-ac2c-bc3815231d52-client-ca-bundle\") pod \"metrics-server-555dfb9dd8-9lkfd\" (UID: \"e8ac9128-16ec-4f9c-ac2c-bc3815231d52\") " pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:15:49.797599 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.797563    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8ac9128-16ec-4f9c-ac2c-bc3815231d52-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-555dfb9dd8-9lkfd\" (UID: \"e8ac9128-16ec-4f9c-ac2c-bc3815231d52\") " pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:15:49.797756 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.797625    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e8ac9128-16ec-4f9c-ac2c-bc3815231d52-metrics-server-audit-profiles\") pod \"metrics-server-555dfb9dd8-9lkfd\" (UID: \"e8ac9128-16ec-4f9c-ac2c-bc3815231d52\") " pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:15:49.797756 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.797654    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e8ac9128-16ec-4f9c-ac2c-bc3815231d52-secret-metrics-server-client-certs\") pod \"metrics-server-555dfb9dd8-9lkfd\" (UID: \"e8ac9128-16ec-4f9c-ac2c-bc3815231d52\") " pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:15:49.797756 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.797675    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e8ac9128-16ec-4f9c-ac2c-bc3815231d52-audit-log\") pod \"metrics-server-555dfb9dd8-9lkfd\" (UID: \"e8ac9128-16ec-4f9c-ac2c-bc3815231d52\") " pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:15:49.797756 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.797712    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e8ac9128-16ec-4f9c-ac2c-bc3815231d52-client-ca-bundle\") pod \"metrics-server-555dfb9dd8-9lkfd\" (UID: \"e8ac9128-16ec-4f9c-ac2c-bc3815231d52\") " pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:15:49.797968 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.797759    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66mbd\" (UniqueName: \"kubernetes.io/projected/e8ac9128-16ec-4f9c-ac2c-bc3815231d52-kube-api-access-66mbd\") pod \"metrics-server-555dfb9dd8-9lkfd\" (UID: \"e8ac9128-16ec-4f9c-ac2c-bc3815231d52\") " pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:15:49.797968 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.797786    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e8ac9128-16ec-4f9c-ac2c-bc3815231d52-secret-metrics-server-tls\") pod \"metrics-server-555dfb9dd8-9lkfd\" (UID: \"e8ac9128-16ec-4f9c-ac2c-bc3815231d52\") " pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:15:49.798471 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.798415    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8ac9128-16ec-4f9c-ac2c-bc3815231d52-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-555dfb9dd8-9lkfd\" (UID: \"e8ac9128-16ec-4f9c-ac2c-bc3815231d52\") " pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:15:49.798687 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.798659    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e8ac9128-16ec-4f9c-ac2c-bc3815231d52-audit-log\") pod \"metrics-server-555dfb9dd8-9lkfd\" (UID: \"e8ac9128-16ec-4f9c-ac2c-bc3815231d52\") " pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:15:49.798854 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.798831    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e8ac9128-16ec-4f9c-ac2c-bc3815231d52-metrics-server-audit-profiles\") pod \"metrics-server-555dfb9dd8-9lkfd\" (UID: \"e8ac9128-16ec-4f9c-ac2c-bc3815231d52\") " pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:15:49.800611 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.800582    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e8ac9128-16ec-4f9c-ac2c-bc3815231d52-secret-metrics-server-tls\") pod \"metrics-server-555dfb9dd8-9lkfd\" (UID: \"e8ac9128-16ec-4f9c-ac2c-bc3815231d52\") " pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:15:49.801007 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.800965    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e8ac9128-16ec-4f9c-ac2c-bc3815231d52-secret-metrics-server-client-certs\") pod \"metrics-server-555dfb9dd8-9lkfd\" (UID: \"e8ac9128-16ec-4f9c-ac2c-bc3815231d52\") " pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:15:49.801458 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.801440    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e8ac9128-16ec-4f9c-ac2c-bc3815231d52-client-ca-bundle\") pod \"metrics-server-555dfb9dd8-9lkfd\" (UID: \"e8ac9128-16ec-4f9c-ac2c-bc3815231d52\") " pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:15:49.807397 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.807374    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66mbd\" (UniqueName: \"kubernetes.io/projected/e8ac9128-16ec-4f9c-ac2c-bc3815231d52-kube-api-access-66mbd\") pod \"metrics-server-555dfb9dd8-9lkfd\" (UID: \"e8ac9128-16ec-4f9c-ac2c-bc3815231d52\") " pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:15:49.959561 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:49.959529    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:15:50.237558 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:50.237532    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"]
Sep 11 12:15:50.239867 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:15:50.239844    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ac9128_16ec_4f9c_ac2c_bc3815231d52.slice/crio-9b263dcadec4e09568be78a7f3c6195a4ee137898c476b340884dfa4f7ef8c15 WatchSource:0}: Error finding container 9b263dcadec4e09568be78a7f3c6195a4ee137898c476b340884dfa4f7ef8c15: Status 404 returned error can't find the container with id 9b263dcadec4e09568be78a7f3c6195a4ee137898c476b340884dfa4f7ef8c15
Sep 11 12:15:50.312471 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:50.312431    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw" event={"ID":"32e2be1f-f47c-4158-bdb3-de6e93f42cf9","Type":"ContainerStarted","Data":"dced4b58c6b61139dde52414dbc9c8fa0691b15286ef26939d031710e322c807"}
Sep 11 12:15:50.312926 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:50.312501    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw" event={"ID":"32e2be1f-f47c-4158-bdb3-de6e93f42cf9","Type":"ContainerStarted","Data":"2c532b8822c2beaea5bf4f7e6b466b58bcb13257ce68d9acc6302ab88264a79c"}
Sep 11 12:15:50.312926 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:50.312536    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw" event={"ID":"32e2be1f-f47c-4158-bdb3-de6e93f42cf9","Type":"ContainerStarted","Data":"a17d3b734f657b8c7a5a01ac70c0aff7ebd785cf51970625ecc18fe4748d9f80"}
Sep 11 12:15:50.313560 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:50.313537    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd" event={"ID":"e8ac9128-16ec-4f9c-ac2c-bc3815231d52","Type":"ContainerStarted","Data":"9b263dcadec4e09568be78a7f3c6195a4ee137898c476b340884dfa4f7ef8c15"}
Sep 11 12:15:50.870098 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:50.870058    2561 patch_prober.go:28] interesting pod/image-registry-6986b74764-wcmn5 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 12:15:50.870279 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:50.870114    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6986b74764-wcmn5" podUID="21a8830b-a81f-488a-bb3e-8ad43abc7d50" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 12:15:51.281733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:51.281704    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ptkj7"
Sep 11 12:15:51.320952 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:51.320922    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw" event={"ID":"32e2be1f-f47c-4158-bdb3-de6e93f42cf9","Type":"ContainerStarted","Data":"ba481728a56005544261bae3e9df06fa650f5c57bd399e8b2fcb0b7600c9c994"}
Sep 11 12:15:52.221797 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:52.221770    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6986b74764-wcmn5"
Sep 11 12:15:52.327750 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:52.327717    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw" event={"ID":"32e2be1f-f47c-4158-bdb3-de6e93f42cf9","Type":"ContainerStarted","Data":"209ec43571a1d6889c5628a16701b2d83c771a8d3b9a6b30138e7a555fe46368"}
Sep 11 12:15:52.328169 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:52.327784    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw" event={"ID":"32e2be1f-f47c-4158-bdb3-de6e93f42cf9","Type":"ContainerStarted","Data":"4e6c38c37a2031fa9378e002ae2e60419961f3734ef85828ce495cd3495a295e"}
Sep 11 12:15:52.328169 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:52.327897    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:52.329416 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:52.329389    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd" event={"ID":"e8ac9128-16ec-4f9c-ac2c-bc3815231d52","Type":"ContainerStarted","Data":"4309a1e5cadc1a4024999fd493b3c6f5c1527effdd16b013191bc35fcd9c9a05"}
Sep 11 12:15:52.358530 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:52.358488    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw" podStartSLOduration=2.215889242 podStartE2EDuration="5.358473497s" totalImagesPullingTime="3.142584255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:15:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:52.357467219 +0000 UTC m=+83.920247008" watchObservedRunningTime="2026-09-11 12:15:52.358473497 +0000 UTC m=+83.921253260"
Sep 11 12:15:52.379433 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:52.379393    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd" podStartSLOduration=1.894157387 podStartE2EDuration="3.379381033s" totalImagesPullingTime="1.485223646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:52.377438338 +0000 UTC m=+83.940218102" watchObservedRunningTime="2026-09-11 12:15:52.379381033 +0000 UTC m=+83.942160796"
Sep 11 12:15:58.340783 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:58.340757    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-59675fdc88-d92dw"
Sep 11 12:15:59.304831 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.304796    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78c6566f55-mh2sp"]
Sep 11 12:15:59.307769 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.307751    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:15:59.310635 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.310614    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-gvxtr\""
Sep 11 12:15:59.310736 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.310613    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 12:15:59.311688 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.311664    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 12:15:59.311688 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.311677    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 12:15:59.311836 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.311693    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 12:15:59.311836 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.311700    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 12:15:59.311836 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.311732    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 12:15:59.311836 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.311671    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 12:15:59.322822 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.322801    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78c6566f55-mh2sp"]
Sep 11 12:15:59.365168 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.365144    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-console-oauth-config\") pod \"console-78c6566f55-mh2sp\" (UID: \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\") " pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:15:59.365501 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.365174    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-console-config\") pod \"console-78c6566f55-mh2sp\" (UID: \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\") " pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:15:59.365501 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.365243    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-oauth-serving-cert\") pod \"console-78c6566f55-mh2sp\" (UID: \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\") " pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:15:59.365501 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.365283    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-console-serving-cert\") pod \"console-78c6566f55-mh2sp\" (UID: \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\") " pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:15:59.365501 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.365330    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngpj5\" (UniqueName: \"kubernetes.io/projected/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-kube-api-access-ngpj5\") pod \"console-78c6566f55-mh2sp\" (UID: \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\") " pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:15:59.365501 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.365351    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-service-ca\") pod \"console-78c6566f55-mh2sp\" (UID: \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\") " pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:15:59.466427 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.466401    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-console-oauth-config\") pod \"console-78c6566f55-mh2sp\" (UID: \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\") " pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:15:59.466532 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.466432    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-console-config\") pod \"console-78c6566f55-mh2sp\" (UID: \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\") " pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:15:59.466532 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.466450    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-oauth-serving-cert\") pod \"console-78c6566f55-mh2sp\" (UID: \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\") " pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:15:59.466532 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.466472    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-console-serving-cert\") pod \"console-78c6566f55-mh2sp\" (UID: \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\") " pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:15:59.466532 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.466507    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngpj5\" (UniqueName: \"kubernetes.io/projected/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-kube-api-access-ngpj5\") pod \"console-78c6566f55-mh2sp\" (UID: \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\") " pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:15:59.466682 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.466535    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-service-ca\") pod \"console-78c6566f55-mh2sp\" (UID: \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\") " pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:15:59.467125 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.467103    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-console-config\") pod \"console-78c6566f55-mh2sp\" (UID: \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\") " pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:15:59.467296 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.467275    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-oauth-serving-cert\") pod \"console-78c6566f55-mh2sp\" (UID: \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\") " pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:15:59.467367 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.467284    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-service-ca\") pod \"console-78c6566f55-mh2sp\" (UID: \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\") " pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:15:59.468964 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.468942    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-console-oauth-config\") pod \"console-78c6566f55-mh2sp\" (UID: \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\") " pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:15:59.469041 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.468952    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-console-serving-cert\") pod \"console-78c6566f55-mh2sp\" (UID: \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\") " pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:15:59.474531 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.474508    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngpj5\" (UniqueName: \"kubernetes.io/projected/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-kube-api-access-ngpj5\") pod \"console-78c6566f55-mh2sp\" (UID: \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\") " pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:15:59.616560 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.616489    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:15:59.890028 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:15:59.889959    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78c6566f55-mh2sp"]
Sep 11 12:15:59.892917 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:15:59.892890    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56f8fb0_69ad_4c8c_87da_82e35b5fd379.slice/crio-f3c427d21db19779b54c4ad0df584a322ac757755fe83c4838ba745b08025366 WatchSource:0}: Error finding container f3c427d21db19779b54c4ad0df584a322ac757755fe83c4838ba745b08025366: Status 404 returned error can't find the container with id f3c427d21db19779b54c4ad0df584a322ac757755fe83c4838ba745b08025366
Sep 11 12:16:00.358878 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:00.358840    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c6566f55-mh2sp" event={"ID":"b56f8fb0-69ad-4c8c-87da-82e35b5fd379","Type":"ContainerStarted","Data":"f3c427d21db19779b54c4ad0df584a322ac757755fe83c4838ba745b08025366"}
Sep 11 12:16:04.376102 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:04.376060    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c6566f55-mh2sp" event={"ID":"b56f8fb0-69ad-4c8c-87da-82e35b5fd379","Type":"ContainerStarted","Data":"ec8ba80b7a559f08d554d962032ceadd3d34b995994920a5ef3bdc6e5c03d88f"}
Sep 11 12:16:04.404793 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:04.404744    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78c6566f55-mh2sp" podStartSLOduration=1.83393309 podStartE2EDuration="5.404729865s" totalImagesPullingTime="3.570796775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:16:04.403095755 +0000 UTC m=+95.965875521" watchObservedRunningTime="2026-09-11 12:16:04.404729865 +0000 UTC m=+95.967509628"
Sep 11 12:16:09.616928 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:09.616892    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:16:09.617310 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:09.616953    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:16:09.621785 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:09.621765    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:16:09.959829 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:09.959750    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:16:09.959829 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:09.959795    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:16:10.401112 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:10.401077    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:16:12.279649 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:12.279619    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kjxhz"
Sep 11 12:16:18.632434 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:18.632400    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78c6566f55-mh2sp"]
Sep 11 12:16:29.965581 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:29.965550    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:16:29.969294 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:29.969273    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-555dfb9dd8-9lkfd"
Sep 11 12:16:43.655960 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:43.655901    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78c6566f55-mh2sp" podUID="b56f8fb0-69ad-4c8c-87da-82e35b5fd379" containerName="console" containerID="cri-o://ec8ba80b7a559f08d554d962032ceadd3d34b995994920a5ef3bdc6e5c03d88f" gracePeriod=15
Sep 11 12:16:43.898260 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:43.898237    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:16:43.991974 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:43.991942    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-service-ca\") pod \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\" (UID: \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\") "
Sep 11 12:16:43.992172 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:43.991992    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngpj5\" (UniqueName: \"kubernetes.io/projected/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-kube-api-access-ngpj5\") pod \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\" (UID: \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\") "
Sep 11 12:16:43.992172 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:43.992018    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-oauth-serving-cert\") pod \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\" (UID: \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\") "
Sep 11 12:16:43.992172 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:43.992051    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-console-serving-cert\") pod \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\" (UID: \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\") "
Sep 11 12:16:43.992172 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:43.992083    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-console-oauth-config\") pod \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\" (UID: \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\") "
Sep 11 12:16:43.992172 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:43.992121    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-console-config\") pod \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\" (UID: \"b56f8fb0-69ad-4c8c-87da-82e35b5fd379\") "
Sep 11 12:16:43.992506 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:43.992465    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b56f8fb0-69ad-4c8c-87da-82e35b5fd379" (UID: "b56f8fb0-69ad-4c8c-87da-82e35b5fd379"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:16:43.992506 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:43.992473    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-service-ca" (OuterVolumeSpecName: "service-ca") pod "b56f8fb0-69ad-4c8c-87da-82e35b5fd379" (UID: "b56f8fb0-69ad-4c8c-87da-82e35b5fd379"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:16:43.992658 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:43.992514    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-console-config" (OuterVolumeSpecName: "console-config") pod "b56f8fb0-69ad-4c8c-87da-82e35b5fd379" (UID: "b56f8fb0-69ad-4c8c-87da-82e35b5fd379"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:16:43.994504 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:43.994475    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b56f8fb0-69ad-4c8c-87da-82e35b5fd379" (UID: "b56f8fb0-69ad-4c8c-87da-82e35b5fd379"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:16:43.994707 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:43.994681    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-kube-api-access-ngpj5" (OuterVolumeSpecName: "kube-api-access-ngpj5") pod "b56f8fb0-69ad-4c8c-87da-82e35b5fd379" (UID: "b56f8fb0-69ad-4c8c-87da-82e35b5fd379"). InnerVolumeSpecName "kube-api-access-ngpj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:16:43.994770 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:43.994689    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b56f8fb0-69ad-4c8c-87da-82e35b5fd379" (UID: "b56f8fb0-69ad-4c8c-87da-82e35b5fd379"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:16:44.093217 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:44.093168    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-service-ca\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:16:44.093217 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:44.093214    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngpj5\" (UniqueName: \"kubernetes.io/projected/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-kube-api-access-ngpj5\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:16:44.093217 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:44.093224    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-oauth-serving-cert\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:16:44.093470 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:44.093234    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-console-serving-cert\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:16:44.093470 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:44.093243    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-console-oauth-config\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:16:44.093470 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:44.093253    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b56f8fb0-69ad-4c8c-87da-82e35b5fd379-console-config\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:16:44.517820 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:44.517784    2561 generic.go:356] "Generic (PLEG): container finished" podID="b56f8fb0-69ad-4c8c-87da-82e35b5fd379" containerID="ec8ba80b7a559f08d554d962032ceadd3d34b995994920a5ef3bdc6e5c03d88f" exitCode=0
Sep 11 12:16:44.517820 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:44.517823    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c6566f55-mh2sp"
Sep 11 12:16:44.518033 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:44.517849    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c6566f55-mh2sp" event={"ID":"b56f8fb0-69ad-4c8c-87da-82e35b5fd379","Type":"ContainerDied","Data":"ec8ba80b7a559f08d554d962032ceadd3d34b995994920a5ef3bdc6e5c03d88f"}
Sep 11 12:16:44.518033 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:44.517895    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c6566f55-mh2sp" event={"ID":"b56f8fb0-69ad-4c8c-87da-82e35b5fd379","Type":"ContainerDied","Data":"f3c427d21db19779b54c4ad0df584a322ac757755fe83c4838ba745b08025366"}
Sep 11 12:16:44.518617 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:44.518594    2561 scope.go:117] "RemoveContainer" containerID="ec8ba80b7a559f08d554d962032ceadd3d34b995994920a5ef3bdc6e5c03d88f"
Sep 11 12:16:44.527661 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:44.527643    2561 scope.go:117] "RemoveContainer" containerID="ec8ba80b7a559f08d554d962032ceadd3d34b995994920a5ef3bdc6e5c03d88f"
Sep 11 12:16:44.527953 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:16:44.527932    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec8ba80b7a559f08d554d962032ceadd3d34b995994920a5ef3bdc6e5c03d88f\": container with ID starting with ec8ba80b7a559f08d554d962032ceadd3d34b995994920a5ef3bdc6e5c03d88f not found: ID does not exist" containerID="ec8ba80b7a559f08d554d962032ceadd3d34b995994920a5ef3bdc6e5c03d88f"
Sep 11 12:16:44.528014 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:44.527960    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec8ba80b7a559f08d554d962032ceadd3d34b995994920a5ef3bdc6e5c03d88f"} err="failed to get container status \"ec8ba80b7a559f08d554d962032ceadd3d34b995994920a5ef3bdc6e5c03d88f\": rpc error: code = NotFound desc = could not find container \"ec8ba80b7a559f08d554d962032ceadd3d34b995994920a5ef3bdc6e5c03d88f\": container with ID starting with ec8ba80b7a559f08d554d962032ceadd3d34b995994920a5ef3bdc6e5c03d88f not found: ID does not exist"
Sep 11 12:16:44.543261 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:44.543239    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78c6566f55-mh2sp"]
Sep 11 12:16:44.547509 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:44.547488    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78c6566f55-mh2sp"]
Sep 11 12:16:44.932884 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:16:44.932812    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b56f8fb0-69ad-4c8c-87da-82e35b5fd379" path="/var/lib/kubelet/pods/b56f8fb0-69ad-4c8c-87da-82e35b5fd379/volumes"
Sep 11 12:17:09.794939 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.794906    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-fcd644666-nqc85"]
Sep 11 12:17:09.795365 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.795157    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b56f8fb0-69ad-4c8c-87da-82e35b5fd379" containerName="console"
Sep 11 12:17:09.795365 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.795167    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b56f8fb0-69ad-4c8c-87da-82e35b5fd379" containerName="console"
Sep 11 12:17:09.795365 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.795244    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b56f8fb0-69ad-4c8c-87da-82e35b5fd379" containerName="console"
Sep 11 12:17:09.798052 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.798031    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:09.805866 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.805845    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 11 12:17:09.805991 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.805969    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 11 12:17:09.806122 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.806107    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-vqg7g\""
Sep 11 12:17:09.806241 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.806223    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 11 12:17:09.806308 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.806224    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 11 12:17:09.806931 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.806913    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 11 12:17:09.815001 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.814985    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 11 12:17:09.825471 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.825444    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-fcd644666-nqc85"]
Sep 11 12:17:09.872264 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.872239    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ead135a5-e594-468d-b742-c6471543656c-secret-telemeter-client\") pod \"telemeter-client-fcd644666-nqc85\" (UID: \"ead135a5-e594-468d-b742-c6471543656c\") " pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:09.872398 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.872280    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ead135a5-e594-468d-b742-c6471543656c-telemeter-client-tls\") pod \"telemeter-client-fcd644666-nqc85\" (UID: \"ead135a5-e594-468d-b742-c6471543656c\") " pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:09.872398 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.872323    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd5rg\" (UniqueName: \"kubernetes.io/projected/ead135a5-e594-468d-b742-c6471543656c-kube-api-access-cd5rg\") pod \"telemeter-client-fcd644666-nqc85\" (UID: \"ead135a5-e594-468d-b742-c6471543656c\") " pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:09.872398 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.872394    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ead135a5-e594-468d-b742-c6471543656c-federate-client-tls\") pod \"telemeter-client-fcd644666-nqc85\" (UID: \"ead135a5-e594-468d-b742-c6471543656c\") " pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:09.872530 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.872424    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ead135a5-e594-468d-b742-c6471543656c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fcd644666-nqc85\" (UID: \"ead135a5-e594-468d-b742-c6471543656c\") " pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:09.872530 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.872454    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ead135a5-e594-468d-b742-c6471543656c-metrics-client-ca\") pod \"telemeter-client-fcd644666-nqc85\" (UID: \"ead135a5-e594-468d-b742-c6471543656c\") " pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:09.872530 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.872498    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ead135a5-e594-468d-b742-c6471543656c-serving-certs-ca-bundle\") pod \"telemeter-client-fcd644666-nqc85\" (UID: \"ead135a5-e594-468d-b742-c6471543656c\") " pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:09.872630 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.872551    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ead135a5-e594-468d-b742-c6471543656c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fcd644666-nqc85\" (UID: \"ead135a5-e594-468d-b742-c6471543656c\") " pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:09.973295 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.973264    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ead135a5-e594-468d-b742-c6471543656c-federate-client-tls\") pod \"telemeter-client-fcd644666-nqc85\" (UID: \"ead135a5-e594-468d-b742-c6471543656c\") " pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:09.973427 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.973300    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ead135a5-e594-468d-b742-c6471543656c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fcd644666-nqc85\" (UID: \"ead135a5-e594-468d-b742-c6471543656c\") " pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:09.973427 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.973319    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ead135a5-e594-468d-b742-c6471543656c-metrics-client-ca\") pod \"telemeter-client-fcd644666-nqc85\" (UID: \"ead135a5-e594-468d-b742-c6471543656c\") " pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:09.973546 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.973436    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ead135a5-e594-468d-b742-c6471543656c-serving-certs-ca-bundle\") pod \"telemeter-client-fcd644666-nqc85\" (UID: \"ead135a5-e594-468d-b742-c6471543656c\") " pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:09.973546 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.973483    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ead135a5-e594-468d-b742-c6471543656c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fcd644666-nqc85\" (UID: \"ead135a5-e594-468d-b742-c6471543656c\") " pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:09.973546 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.973514    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ead135a5-e594-468d-b742-c6471543656c-secret-telemeter-client\") pod \"telemeter-client-fcd644666-nqc85\" (UID: \"ead135a5-e594-468d-b742-c6471543656c\") " pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:09.973686 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.973553    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ead135a5-e594-468d-b742-c6471543656c-telemeter-client-tls\") pod \"telemeter-client-fcd644666-nqc85\" (UID: \"ead135a5-e594-468d-b742-c6471543656c\") " pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:09.973686 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.973593    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd5rg\" (UniqueName: \"kubernetes.io/projected/ead135a5-e594-468d-b742-c6471543656c-kube-api-access-cd5rg\") pod \"telemeter-client-fcd644666-nqc85\" (UID: \"ead135a5-e594-468d-b742-c6471543656c\") " pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:09.973985 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.973959    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ead135a5-e594-468d-b742-c6471543656c-metrics-client-ca\") pod \"telemeter-client-fcd644666-nqc85\" (UID: \"ead135a5-e594-468d-b742-c6471543656c\") " pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:09.974127 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.974099    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ead135a5-e594-468d-b742-c6471543656c-serving-certs-ca-bundle\") pod \"telemeter-client-fcd644666-nqc85\" (UID: \"ead135a5-e594-468d-b742-c6471543656c\") " pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:09.974686 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.974661    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ead135a5-e594-468d-b742-c6471543656c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-fcd644666-nqc85\" (UID: \"ead135a5-e594-468d-b742-c6471543656c\") " pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:09.975805 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.975784    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ead135a5-e594-468d-b742-c6471543656c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-fcd644666-nqc85\" (UID: \"ead135a5-e594-468d-b742-c6471543656c\") " pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:09.976017 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.975998    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ead135a5-e594-468d-b742-c6471543656c-federate-client-tls\") pod \"telemeter-client-fcd644666-nqc85\" (UID: \"ead135a5-e594-468d-b742-c6471543656c\") " pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:09.976076 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.976041    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ead135a5-e594-468d-b742-c6471543656c-telemeter-client-tls\") pod \"telemeter-client-fcd644666-nqc85\" (UID: \"ead135a5-e594-468d-b742-c6471543656c\") " pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:09.976112 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.976089    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ead135a5-e594-468d-b742-c6471543656c-secret-telemeter-client\") pod \"telemeter-client-fcd644666-nqc85\" (UID: \"ead135a5-e594-468d-b742-c6471543656c\") " pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:09.981798 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:09.981775    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd5rg\" (UniqueName: \"kubernetes.io/projected/ead135a5-e594-468d-b742-c6471543656c-kube-api-access-cd5rg\") pod \"telemeter-client-fcd644666-nqc85\" (UID: \"ead135a5-e594-468d-b742-c6471543656c\") " pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:10.107029 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:10.106934    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-fcd644666-nqc85"
Sep 11 12:17:10.393950 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:10.393928    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-fcd644666-nqc85"]
Sep 11 12:17:10.395619 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:17:10.395590    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead135a5_e594_468d_b742_c6471543656c.slice/crio-fb0c77e1ec224e4e26e1b7b25266806fc476580e168ee0f4fcb49605626852f0 WatchSource:0}: Error finding container fb0c77e1ec224e4e26e1b7b25266806fc476580e168ee0f4fcb49605626852f0: Status 404 returned error can't find the container with id fb0c77e1ec224e4e26e1b7b25266806fc476580e168ee0f4fcb49605626852f0
Sep 11 12:17:10.612073 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:10.612041    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fcd644666-nqc85" event={"ID":"ead135a5-e594-468d-b742-c6471543656c","Type":"ContainerStarted","Data":"fb0c77e1ec224e4e26e1b7b25266806fc476580e168ee0f4fcb49605626852f0"}
Sep 11 12:17:12.620370 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:12.620294    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fcd644666-nqc85" event={"ID":"ead135a5-e594-468d-b742-c6471543656c","Type":"ContainerStarted","Data":"0d060271b5e19e1ad02f2366213fbff204e9dba76c8ee8675efcac4c4f805e48"}
Sep 11 12:17:13.625412 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:13.625387    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fcd644666-nqc85" event={"ID":"ead135a5-e594-468d-b742-c6471543656c","Type":"ContainerStarted","Data":"f83c345ecc4ca563f6eb53df891e3ca5b5c08c82dfeaa7ba66d16292036f4166"}
Sep 11 12:17:14.631076 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:14.631039    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-fcd644666-nqc85" event={"ID":"ead135a5-e594-468d-b742-c6471543656c","Type":"ContainerStarted","Data":"72784996a7fe98a6c05104105e7545e8f815e3ab4c435e73ddbf1e4031f9a193"}
Sep 11 12:17:14.657299 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:17:14.657243    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-fcd644666-nqc85" podStartSLOduration=2.5928013439999997 podStartE2EDuration="5.657228323s" totalImagesPullingTime="3.064426979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:17:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:17:14.656875416 +0000 UTC m=+166.219655214" watchObservedRunningTime="2026-09-11 12:17:14.657228323 +0000 UTC m=+166.220008085"
Sep 11 12:18:33.695449 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.695412    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86589f6569-rmd2g"]
Sep 11 12:18:33.697575 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.697558    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:33.700377 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.700349    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 12:18:33.700517 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.700394    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 12:18:33.700517 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.700405    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 12:18:33.700517 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.700419    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 12:18:33.700821 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.700805    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-gvxtr\""
Sep 11 12:18:33.700894 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.700806    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 12:18:33.701277 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.701258    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 12:18:33.701351 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.701259    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 12:18:33.708438 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.708411    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 12:18:33.713294 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.713267    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86589f6569-rmd2g"]
Sep 11 12:18:33.776825 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.776796    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92fe1048-0690-4d1e-bced-15ea06898832-console-config\") pod \"console-86589f6569-rmd2g\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") " pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:33.776970 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.776830    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92fe1048-0690-4d1e-bced-15ea06898832-oauth-serving-cert\") pod \"console-86589f6569-rmd2g\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") " pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:33.776970 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.776850    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92fe1048-0690-4d1e-bced-15ea06898832-console-serving-cert\") pod \"console-86589f6569-rmd2g\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") " pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:33.776970 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.776870    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92fe1048-0690-4d1e-bced-15ea06898832-trusted-ca-bundle\") pod \"console-86589f6569-rmd2g\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") " pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:33.776970 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.776943    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92fe1048-0690-4d1e-bced-15ea06898832-service-ca\") pod \"console-86589f6569-rmd2g\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") " pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:33.777109 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.776989    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92fe1048-0690-4d1e-bced-15ea06898832-console-oauth-config\") pod \"console-86589f6569-rmd2g\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") " pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:33.777109 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.777010    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6jv4\" (UniqueName: \"kubernetes.io/projected/92fe1048-0690-4d1e-bced-15ea06898832-kube-api-access-n6jv4\") pod \"console-86589f6569-rmd2g\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") " pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:33.878181 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.878141    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92fe1048-0690-4d1e-bced-15ea06898832-oauth-serving-cert\") pod \"console-86589f6569-rmd2g\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") " pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:33.878181 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.878200    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92fe1048-0690-4d1e-bced-15ea06898832-console-serving-cert\") pod \"console-86589f6569-rmd2g\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") " pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:33.878371 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.878223    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92fe1048-0690-4d1e-bced-15ea06898832-trusted-ca-bundle\") pod \"console-86589f6569-rmd2g\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") " pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:33.878371 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.878243    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92fe1048-0690-4d1e-bced-15ea06898832-service-ca\") pod \"console-86589f6569-rmd2g\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") " pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:33.878371 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.878272    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92fe1048-0690-4d1e-bced-15ea06898832-console-oauth-config\") pod \"console-86589f6569-rmd2g\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") " pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:33.878371 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.878291    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6jv4\" (UniqueName: \"kubernetes.io/projected/92fe1048-0690-4d1e-bced-15ea06898832-kube-api-access-n6jv4\") pod \"console-86589f6569-rmd2g\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") " pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:33.878371 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.878333    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92fe1048-0690-4d1e-bced-15ea06898832-console-config\") pod \"console-86589f6569-rmd2g\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") " pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:33.878969 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.878941    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92fe1048-0690-4d1e-bced-15ea06898832-oauth-serving-cert\") pod \"console-86589f6569-rmd2g\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") " pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:33.879129 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.879007    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92fe1048-0690-4d1e-bced-15ea06898832-console-config\") pod \"console-86589f6569-rmd2g\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") " pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:33.879250 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.879018    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92fe1048-0690-4d1e-bced-15ea06898832-service-ca\") pod \"console-86589f6569-rmd2g\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") " pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:33.879250 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.879074    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92fe1048-0690-4d1e-bced-15ea06898832-trusted-ca-bundle\") pod \"console-86589f6569-rmd2g\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") " pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:33.880709 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.880686    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92fe1048-0690-4d1e-bced-15ea06898832-console-oauth-config\") pod \"console-86589f6569-rmd2g\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") " pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:33.880801 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.880781    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92fe1048-0690-4d1e-bced-15ea06898832-console-serving-cert\") pod \"console-86589f6569-rmd2g\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") " pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:33.887316 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:33.887296    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6jv4\" (UniqueName: \"kubernetes.io/projected/92fe1048-0690-4d1e-bced-15ea06898832-kube-api-access-n6jv4\") pod \"console-86589f6569-rmd2g\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") " pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:34.010865 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:34.010842    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:34.283605 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:34.283534    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86589f6569-rmd2g"]
Sep 11 12:18:34.289297 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:18:34.289270    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92fe1048_0690_4d1e_bced_15ea06898832.slice/crio-c7396559c2b1b6385e39acf195ab85c3f3ffa16e6b2550b6085bbbe3a2c90a74 WatchSource:0}: Error finding container c7396559c2b1b6385e39acf195ab85c3f3ffa16e6b2550b6085bbbe3a2c90a74: Status 404 returned error can't find the container with id c7396559c2b1b6385e39acf195ab85c3f3ffa16e6b2550b6085bbbe3a2c90a74
Sep 11 12:18:34.918110 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:34.918072    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86589f6569-rmd2g" event={"ID":"92fe1048-0690-4d1e-bced-15ea06898832","Type":"ContainerStarted","Data":"2b6ad77ad07ec57ee29082f838be1cd553c643d09a977b5af173ae6a27052965"}
Sep 11 12:18:34.918572 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:34.918127    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86589f6569-rmd2g" event={"ID":"92fe1048-0690-4d1e-bced-15ea06898832","Type":"ContainerStarted","Data":"c7396559c2b1b6385e39acf195ab85c3f3ffa16e6b2550b6085bbbe3a2c90a74"}
Sep 11 12:18:34.950353 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:34.950305    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86589f6569-rmd2g" podStartSLOduration=1.950285811 podStartE2EDuration="1.950285811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:18:34.949804275 +0000 UTC m=+246.512584038" watchObservedRunningTime="2026-09-11 12:18:34.950285811 +0000 UTC m=+246.513065633"
Sep 11 12:18:44.011716 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:44.011682    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:44.012113 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:44.011802    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:44.016585 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:44.016561    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:18:44.956229 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:18:44.956201    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:19:28.815552 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:19:28.815519    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcfm6_88402687-4e5b-411c-be7a-4cb955f90425/ovn-acl-logging/0.log"
Sep 11 12:19:28.819175 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:19:28.819153    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcfm6_88402687-4e5b-411c-be7a-4cb955f90425/ovn-acl-logging/0.log"
Sep 11 12:19:28.820716 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:19:28.820695    2561 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 12:20:12.914346 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:12.914312    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ks2fb"]
Sep 11 12:20:12.917292 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:12.917264    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ks2fb"
Sep 11 12:20:12.919849 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:12.919825    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 12:20:12.919957 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:12.919857    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-dt25g\""
Sep 11 12:20:12.919957 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:12.919857    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 12:20:12.930396 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:12.930378    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ks2fb"]
Sep 11 12:20:13.028990 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:13.028954    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbp5b\" (UniqueName: \"kubernetes.io/projected/067b735b-ebd4-4e90-b24f-f3907e2a4496-kube-api-access-xbp5b\") pod \"cert-manager-cainjector-559cdd8c87-ks2fb\" (UID: \"067b735b-ebd4-4e90-b24f-f3907e2a4496\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ks2fb"
Sep 11 12:20:13.029129 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:13.029033    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/067b735b-ebd4-4e90-b24f-f3907e2a4496-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ks2fb\" (UID: \"067b735b-ebd4-4e90-b24f-f3907e2a4496\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ks2fb"
Sep 11 12:20:13.130317 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:13.130291    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbp5b\" (UniqueName: \"kubernetes.io/projected/067b735b-ebd4-4e90-b24f-f3907e2a4496-kube-api-access-xbp5b\") pod \"cert-manager-cainjector-559cdd8c87-ks2fb\" (UID: \"067b735b-ebd4-4e90-b24f-f3907e2a4496\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ks2fb"
Sep 11 12:20:13.130405 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:13.130344    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/067b735b-ebd4-4e90-b24f-f3907e2a4496-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ks2fb\" (UID: \"067b735b-ebd4-4e90-b24f-f3907e2a4496\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ks2fb"
Sep 11 12:20:13.140153 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:13.140123    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/067b735b-ebd4-4e90-b24f-f3907e2a4496-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ks2fb\" (UID: \"067b735b-ebd4-4e90-b24f-f3907e2a4496\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ks2fb"
Sep 11 12:20:13.140430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:13.140410    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbp5b\" (UniqueName: \"kubernetes.io/projected/067b735b-ebd4-4e90-b24f-f3907e2a4496-kube-api-access-xbp5b\") pod \"cert-manager-cainjector-559cdd8c87-ks2fb\" (UID: \"067b735b-ebd4-4e90-b24f-f3907e2a4496\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ks2fb"
Sep 11 12:20:13.226365 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:13.226295    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ks2fb"
Sep 11 12:20:13.494259 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:13.494228    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ks2fb"]
Sep 11 12:20:13.496665 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:20:13.496631    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod067b735b_ebd4_4e90_b24f_f3907e2a4496.slice/crio-271980022bed0554817d8a7516d202b7ae215ca0a14d0ac6f97c9fe571980598 WatchSource:0}: Error finding container 271980022bed0554817d8a7516d202b7ae215ca0a14d0ac6f97c9fe571980598: Status 404 returned error can't find the container with id 271980022bed0554817d8a7516d202b7ae215ca0a14d0ac6f97c9fe571980598
Sep 11 12:20:13.498673 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:13.498658    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 12:20:14.268946 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:14.268903    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ks2fb" event={"ID":"067b735b-ebd4-4e90-b24f-f3907e2a4496","Type":"ContainerStarted","Data":"271980022bed0554817d8a7516d202b7ae215ca0a14d0ac6f97c9fe571980598"}
Sep 11 12:20:17.282155 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:17.282120    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ks2fb" event={"ID":"067b735b-ebd4-4e90-b24f-f3907e2a4496","Type":"ContainerStarted","Data":"8dd673e75edd51083650ec7027207c6a0c41ee02ff0c76f0cb26686f786c7064"}
Sep 11 12:20:17.298247 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:17.298185    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ks2fb" podStartSLOduration=2.372864909 podStartE2EDuration="5.298170368s" totalImagesPullingTime="2.925305459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:20:17.297226629 +0000 UTC m=+348.860006393" watchObservedRunningTime="2026-09-11 12:20:17.298170368 +0000 UTC m=+348.860950131"
Sep 11 12:20:39.978628 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:39.978547    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg"]
Sep 11 12:20:39.983050 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:39.983028    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg"
Sep 11 12:20:39.985789 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:39.985767    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 12:20:39.985911 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:39.985777    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 12:20:39.985973 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:39.985950    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 12:20:39.986051 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:39.986029    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-n5jwr\""
Sep 11 12:20:39.986176 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:39.986140    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 12:20:40.003448 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:40.003420    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg"]
Sep 11 12:20:40.143210 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:40.143161    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db95a0f5-e92b-4bed-b625-c402af9c93d0-tmp\") pod \"opendatahub-operator-controller-manager-8dd64745b-xxnpg\" (UID: \"db95a0f5-e92b-4bed-b625-c402af9c93d0\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg"
Sep 11 12:20:40.143374 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:40.143220    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db95a0f5-e92b-4bed-b625-c402af9c93d0-webhook-cert\") pod \"opendatahub-operator-controller-manager-8dd64745b-xxnpg\" (UID: \"db95a0f5-e92b-4bed-b625-c402af9c93d0\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg"
Sep 11 12:20:40.143374 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:40.143276    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/db95a0f5-e92b-4bed-b625-c402af9c93d0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8dd64745b-xxnpg\" (UID: \"db95a0f5-e92b-4bed-b625-c402af9c93d0\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg"
Sep 11 12:20:40.143374 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:40.143318    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/db95a0f5-e92b-4bed-b625-c402af9c93d0-manifests\") pod \"opendatahub-operator-controller-manager-8dd64745b-xxnpg\" (UID: \"db95a0f5-e92b-4bed-b625-c402af9c93d0\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg"
Sep 11 12:20:40.143374 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:40.143344    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2q6b\" (UniqueName: \"kubernetes.io/projected/db95a0f5-e92b-4bed-b625-c402af9c93d0-kube-api-access-j2q6b\") pod \"opendatahub-operator-controller-manager-8dd64745b-xxnpg\" (UID: \"db95a0f5-e92b-4bed-b625-c402af9c93d0\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg"
Sep 11 12:20:40.244399 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:40.244305    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2q6b\" (UniqueName: \"kubernetes.io/projected/db95a0f5-e92b-4bed-b625-c402af9c93d0-kube-api-access-j2q6b\") pod \"opendatahub-operator-controller-manager-8dd64745b-xxnpg\" (UID: \"db95a0f5-e92b-4bed-b625-c402af9c93d0\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg"
Sep 11 12:20:40.244399 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:40.244368    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db95a0f5-e92b-4bed-b625-c402af9c93d0-tmp\") pod \"opendatahub-operator-controller-manager-8dd64745b-xxnpg\" (UID: \"db95a0f5-e92b-4bed-b625-c402af9c93d0\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg"
Sep 11 12:20:40.244399 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:40.244391    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db95a0f5-e92b-4bed-b625-c402af9c93d0-webhook-cert\") pod \"opendatahub-operator-controller-manager-8dd64745b-xxnpg\" (UID: \"db95a0f5-e92b-4bed-b625-c402af9c93d0\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg"
Sep 11 12:20:40.244676 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:40.244418    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/db95a0f5-e92b-4bed-b625-c402af9c93d0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8dd64745b-xxnpg\" (UID: \"db95a0f5-e92b-4bed-b625-c402af9c93d0\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg"
Sep 11 12:20:40.244676 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:40.244444    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/db95a0f5-e92b-4bed-b625-c402af9c93d0-manifests\") pod \"opendatahub-operator-controller-manager-8dd64745b-xxnpg\" (UID: \"db95a0f5-e92b-4bed-b625-c402af9c93d0\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg"
Sep 11 12:20:40.244841 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:40.244819    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/db95a0f5-e92b-4bed-b625-c402af9c93d0-manifests\") pod \"opendatahub-operator-controller-manager-8dd64745b-xxnpg\" (UID: \"db95a0f5-e92b-4bed-b625-c402af9c93d0\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg"
Sep 11 12:20:40.246699 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:40.246667    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db95a0f5-e92b-4bed-b625-c402af9c93d0-tmp\") pod \"opendatahub-operator-controller-manager-8dd64745b-xxnpg\" (UID: \"db95a0f5-e92b-4bed-b625-c402af9c93d0\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg"
Sep 11 12:20:40.246813 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:40.246795    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db95a0f5-e92b-4bed-b625-c402af9c93d0-webhook-cert\") pod \"opendatahub-operator-controller-manager-8dd64745b-xxnpg\" (UID: \"db95a0f5-e92b-4bed-b625-c402af9c93d0\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg"
Sep 11 12:20:40.246903 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:40.246888    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/db95a0f5-e92b-4bed-b625-c402af9c93d0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8dd64745b-xxnpg\" (UID: \"db95a0f5-e92b-4bed-b625-c402af9c93d0\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg"
Sep 11 12:20:40.252391 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:40.252372    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2q6b\" (UniqueName: \"kubernetes.io/projected/db95a0f5-e92b-4bed-b625-c402af9c93d0-kube-api-access-j2q6b\") pod \"opendatahub-operator-controller-manager-8dd64745b-xxnpg\" (UID: \"db95a0f5-e92b-4bed-b625-c402af9c93d0\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg"
Sep 11 12:20:40.293165 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:40.293139    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg"
Sep 11 12:20:40.566444 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:40.566233    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg"]
Sep 11 12:20:40.569462 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:20:40.569433    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb95a0f5_e92b_4bed_b625_c402af9c93d0.slice/crio-e3ca0f17f1c014ef745f30ea3a7fe4fc2b3ca2a58e567ba1c055887a00e5b327 WatchSource:0}: Error finding container e3ca0f17f1c014ef745f30ea3a7fe4fc2b3ca2a58e567ba1c055887a00e5b327: Status 404 returned error can't find the container with id e3ca0f17f1c014ef745f30ea3a7fe4fc2b3ca2a58e567ba1c055887a00e5b327
Sep 11 12:20:41.372376 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:41.372335    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg" event={"ID":"db95a0f5-e92b-4bed-b625-c402af9c93d0","Type":"ContainerStarted","Data":"e3ca0f17f1c014ef745f30ea3a7fe4fc2b3ca2a58e567ba1c055887a00e5b327"}
Sep 11 12:20:44.390827 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:44.390792    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg" event={"ID":"db95a0f5-e92b-4bed-b625-c402af9c93d0","Type":"ContainerStarted","Data":"27e14e38075b23557b12d3177350d6c9c8769451524ac1e0dcb07b24bbbf71ae"}
Sep 11 12:20:45.396239 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:45.396206    2561 generic.go:356] "Generic (PLEG): container finished" podID="db95a0f5-e92b-4bed-b625-c402af9c93d0" containerID="27e14e38075b23557b12d3177350d6c9c8769451524ac1e0dcb07b24bbbf71ae" exitCode=0
Sep 11 12:20:45.396650 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:45.396292    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg" event={"ID":"db95a0f5-e92b-4bed-b625-c402af9c93d0","Type":"ContainerDied","Data":"27e14e38075b23557b12d3177350d6c9c8769451524ac1e0dcb07b24bbbf71ae"}
Sep 11 12:20:46.401807 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:46.401771    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg" event={"ID":"db95a0f5-e92b-4bed-b625-c402af9c93d0","Type":"ContainerStarted","Data":"be8b4ac0ec6a24b80de2aaf1625bb4fe5b2f0b5ac58d6a0af4eeb3df4fefc274"}
Sep 11 12:20:46.402296 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:46.401892    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg"
Sep 11 12:20:46.425756 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:46.425710    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg" podStartSLOduration=3.048498909 podStartE2EDuration="7.425696271s" totalImagesPullingTime="3.330903703s" totalInitContainerRuntime="1.046293659s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:20:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:20:46.423839278 +0000 UTC m=+377.986619041" watchObservedRunningTime="2026-09-11 12:20:46.425696271 +0000 UTC m=+377.988476033"
Sep 11 12:20:57.408153 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:57.408118    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-xxnpg"
Sep 11 12:20:59.910397 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:59.910367    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67fcb57878-lbbjf"]
Sep 11 12:20:59.913170 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:59.913154    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:20:59.935336 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:59.935308    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67fcb57878-lbbjf"]
Sep 11 12:20:59.994095 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:59.994071    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5b5f595c8f-7cjqr"]
Sep 11 12:20:59.997185 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:20:59.997165    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-7cjqr"
Sep 11 12:21:00.001208 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.001169    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 12:21:00.001591 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.001574    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 12:21:00.002055 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.002032    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 12:21:00.002245 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.002228    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 12:21:00.002913 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.002894    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5v7l\" (UniqueName: \"kubernetes.io/projected/fbf39cdb-14b2-49a7-a2c1-145223f7588a-kube-api-access-v5v7l\") pod \"console-67fcb57878-lbbjf\" (UID: \"fbf39cdb-14b2-49a7-a2c1-145223f7588a\") " pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:00.003001 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.002928    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbf39cdb-14b2-49a7-a2c1-145223f7588a-trusted-ca-bundle\") pod \"console-67fcb57878-lbbjf\" (UID: \"fbf39cdb-14b2-49a7-a2c1-145223f7588a\") " pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:00.003161 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.003011    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fbf39cdb-14b2-49a7-a2c1-145223f7588a-console-oauth-config\") pod \"console-67fcb57878-lbbjf\" (UID: \"fbf39cdb-14b2-49a7-a2c1-145223f7588a\") " pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:00.003268 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.003162    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fbf39cdb-14b2-49a7-a2c1-145223f7588a-console-serving-cert\") pod \"console-67fcb57878-lbbjf\" (UID: \"fbf39cdb-14b2-49a7-a2c1-145223f7588a\") " pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:00.003268 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.003234    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fbf39cdb-14b2-49a7-a2c1-145223f7588a-oauth-serving-cert\") pod \"console-67fcb57878-lbbjf\" (UID: \"fbf39cdb-14b2-49a7-a2c1-145223f7588a\") " pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:00.003368 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.003278    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fbf39cdb-14b2-49a7-a2c1-145223f7588a-console-config\") pod \"console-67fcb57878-lbbjf\" (UID: \"fbf39cdb-14b2-49a7-a2c1-145223f7588a\") " pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:00.003411 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.003381    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf39cdb-14b2-49a7-a2c1-145223f7588a-service-ca\") pod \"console-67fcb57878-lbbjf\" (UID: \"fbf39cdb-14b2-49a7-a2c1-145223f7588a\") " pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:00.018516 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.018480    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b5f595c8f-7cjqr"]
Sep 11 12:21:00.104322 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.104287    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5v7l\" (UniqueName: \"kubernetes.io/projected/fbf39cdb-14b2-49a7-a2c1-145223f7588a-kube-api-access-v5v7l\") pod \"console-67fcb57878-lbbjf\" (UID: \"fbf39cdb-14b2-49a7-a2c1-145223f7588a\") " pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:00.104322 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.104323    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbf39cdb-14b2-49a7-a2c1-145223f7588a-trusted-ca-bundle\") pod \"console-67fcb57878-lbbjf\" (UID: \"fbf39cdb-14b2-49a7-a2c1-145223f7588a\") " pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:00.104547 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.104351    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fbf39cdb-14b2-49a7-a2c1-145223f7588a-console-oauth-config\") pod \"console-67fcb57878-lbbjf\" (UID: \"fbf39cdb-14b2-49a7-a2c1-145223f7588a\") " pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:00.104547 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.104482    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f25fe8ac-8170-4f84-a77c-8f077fe8b458-tls-certs\") pod \"kube-auth-proxy-5b5f595c8f-7cjqr\" (UID: \"f25fe8ac-8170-4f84-a77c-8f077fe8b458\") " pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-7cjqr"
Sep 11 12:21:00.104547 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.104532    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f25fe8ac-8170-4f84-a77c-8f077fe8b458-tmp\") pod \"kube-auth-proxy-5b5f595c8f-7cjqr\" (UID: \"f25fe8ac-8170-4f84-a77c-8f077fe8b458\") " pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-7cjqr"
Sep 11 12:21:00.104652 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.104595    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v6tf\" (UniqueName: \"kubernetes.io/projected/f25fe8ac-8170-4f84-a77c-8f077fe8b458-kube-api-access-8v6tf\") pod \"kube-auth-proxy-5b5f595c8f-7cjqr\" (UID: \"f25fe8ac-8170-4f84-a77c-8f077fe8b458\") " pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-7cjqr"
Sep 11 12:21:00.104652 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.104645    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fbf39cdb-14b2-49a7-a2c1-145223f7588a-console-serving-cert\") pod \"console-67fcb57878-lbbjf\" (UID: \"fbf39cdb-14b2-49a7-a2c1-145223f7588a\") " pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:00.104725 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.104669    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fbf39cdb-14b2-49a7-a2c1-145223f7588a-oauth-serving-cert\") pod \"console-67fcb57878-lbbjf\" (UID: \"fbf39cdb-14b2-49a7-a2c1-145223f7588a\") " pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:00.104806 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.104788    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fbf39cdb-14b2-49a7-a2c1-145223f7588a-console-config\") pod \"console-67fcb57878-lbbjf\" (UID: \"fbf39cdb-14b2-49a7-a2c1-145223f7588a\") " pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:00.104891 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.104839    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf39cdb-14b2-49a7-a2c1-145223f7588a-service-ca\") pod \"console-67fcb57878-lbbjf\" (UID: \"fbf39cdb-14b2-49a7-a2c1-145223f7588a\") " pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:00.105245 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.105217    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbf39cdb-14b2-49a7-a2c1-145223f7588a-trusted-ca-bundle\") pod \"console-67fcb57878-lbbjf\" (UID: \"fbf39cdb-14b2-49a7-a2c1-145223f7588a\") " pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:00.105453 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.105434    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fbf39cdb-14b2-49a7-a2c1-145223f7588a-console-config\") pod \"console-67fcb57878-lbbjf\" (UID: \"fbf39cdb-14b2-49a7-a2c1-145223f7588a\") " pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:00.105453 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.105440    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fbf39cdb-14b2-49a7-a2c1-145223f7588a-oauth-serving-cert\") pod \"console-67fcb57878-lbbjf\" (UID: \"fbf39cdb-14b2-49a7-a2c1-145223f7588a\") " pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:00.105601 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.105571    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf39cdb-14b2-49a7-a2c1-145223f7588a-service-ca\") pod \"console-67fcb57878-lbbjf\" (UID: \"fbf39cdb-14b2-49a7-a2c1-145223f7588a\") " pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:00.106828 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.106802    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fbf39cdb-14b2-49a7-a2c1-145223f7588a-console-oauth-config\") pod \"console-67fcb57878-lbbjf\" (UID: \"fbf39cdb-14b2-49a7-a2c1-145223f7588a\") " pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:00.107016 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.106999    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fbf39cdb-14b2-49a7-a2c1-145223f7588a-console-serving-cert\") pod \"console-67fcb57878-lbbjf\" (UID: \"fbf39cdb-14b2-49a7-a2c1-145223f7588a\") " pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:00.127850 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.127827    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5v7l\" (UniqueName: \"kubernetes.io/projected/fbf39cdb-14b2-49a7-a2c1-145223f7588a-kube-api-access-v5v7l\") pod \"console-67fcb57878-lbbjf\" (UID: \"fbf39cdb-14b2-49a7-a2c1-145223f7588a\") " pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:00.206071 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.206009    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f25fe8ac-8170-4f84-a77c-8f077fe8b458-tls-certs\") pod \"kube-auth-proxy-5b5f595c8f-7cjqr\" (UID: \"f25fe8ac-8170-4f84-a77c-8f077fe8b458\") " pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-7cjqr"
Sep 11 12:21:00.206071 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.206038    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f25fe8ac-8170-4f84-a77c-8f077fe8b458-tmp\") pod \"kube-auth-proxy-5b5f595c8f-7cjqr\" (UID: \"f25fe8ac-8170-4f84-a77c-8f077fe8b458\") " pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-7cjqr"
Sep 11 12:21:00.206071 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.206064    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v6tf\" (UniqueName: \"kubernetes.io/projected/f25fe8ac-8170-4f84-a77c-8f077fe8b458-kube-api-access-8v6tf\") pod \"kube-auth-proxy-5b5f595c8f-7cjqr\" (UID: \"f25fe8ac-8170-4f84-a77c-8f077fe8b458\") " pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-7cjqr"
Sep 11 12:21:00.206257 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:21:00.206238    2561 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 11 12:21:00.206327 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:21:00.206317    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f25fe8ac-8170-4f84-a77c-8f077fe8b458-tls-certs podName:f25fe8ac-8170-4f84-a77c-8f077fe8b458 nodeName:}" failed. No retries permitted until 2026-09-11 12:21:00.706299085 +0000 UTC m=+392.269078840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f25fe8ac-8170-4f84-a77c-8f077fe8b458-tls-certs") pod "kube-auth-proxy-5b5f595c8f-7cjqr" (UID: "f25fe8ac-8170-4f84-a77c-8f077fe8b458") : secret "kube-auth-proxy-tls" not found
Sep 11 12:21:00.208003 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.207980    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f25fe8ac-8170-4f84-a77c-8f077fe8b458-tmp\") pod \"kube-auth-proxy-5b5f595c8f-7cjqr\" (UID: \"f25fe8ac-8170-4f84-a77c-8f077fe8b458\") " pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-7cjqr"
Sep 11 12:21:00.221978 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.221953    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:00.230391 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.230367    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v6tf\" (UniqueName: \"kubernetes.io/projected/f25fe8ac-8170-4f84-a77c-8f077fe8b458-kube-api-access-8v6tf\") pod \"kube-auth-proxy-5b5f595c8f-7cjqr\" (UID: \"f25fe8ac-8170-4f84-a77c-8f077fe8b458\") " pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-7cjqr"
Sep 11 12:21:00.533141 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.533117    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67fcb57878-lbbjf"]
Sep 11 12:21:00.536159 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:21:00.536127    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf39cdb_14b2_49a7_a2c1_145223f7588a.slice/crio-9aec18ee3862d442d3046013ef21c6389a65c32caf8c8a0b8553da28926150b0 WatchSource:0}: Error finding container 9aec18ee3862d442d3046013ef21c6389a65c32caf8c8a0b8553da28926150b0: Status 404 returned error can't find the container with id 9aec18ee3862d442d3046013ef21c6389a65c32caf8c8a0b8553da28926150b0
Sep 11 12:21:00.711326 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.711294    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f25fe8ac-8170-4f84-a77c-8f077fe8b458-tls-certs\") pod \"kube-auth-proxy-5b5f595c8f-7cjqr\" (UID: \"f25fe8ac-8170-4f84-a77c-8f077fe8b458\") " pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-7cjqr"
Sep 11 12:21:00.713593 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.713572    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f25fe8ac-8170-4f84-a77c-8f077fe8b458-tls-certs\") pod \"kube-auth-proxy-5b5f595c8f-7cjqr\" (UID: \"f25fe8ac-8170-4f84-a77c-8f077fe8b458\") " pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-7cjqr"
Sep 11 12:21:00.906006 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:00.905928    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-7cjqr"
Sep 11 12:21:01.185808 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.185606    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b5f595c8f-7cjqr"]
Sep 11 12:21:01.188246 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:21:01.188213    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25fe8ac_8170_4f84_a77c_8f077fe8b458.slice/crio-a44a0390f9c6594d9cda6724f8ce55c9e893334b0682ba65f19f2dbddd82c1ea WatchSource:0}: Error finding container a44a0390f9c6594d9cda6724f8ce55c9e893334b0682ba65f19f2dbddd82c1ea: Status 404 returned error can't find the container with id a44a0390f9c6594d9cda6724f8ce55c9e893334b0682ba65f19f2dbddd82c1ea
Sep 11 12:21:01.466913 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.466823    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-7cjqr" event={"ID":"f25fe8ac-8170-4f84-a77c-8f077fe8b458","Type":"ContainerStarted","Data":"a44a0390f9c6594d9cda6724f8ce55c9e893334b0682ba65f19f2dbddd82c1ea"}
Sep 11 12:21:01.468343 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.468317    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67fcb57878-lbbjf" event={"ID":"fbf39cdb-14b2-49a7-a2c1-145223f7588a","Type":"ContainerStarted","Data":"9843c276c5ffc933024b113f1405ca22732baaf30062b2a15309bd0409c68f6e"}
Sep 11 12:21:01.468472 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.468356    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67fcb57878-lbbjf" event={"ID":"fbf39cdb-14b2-49a7-a2c1-145223f7588a","Type":"ContainerStarted","Data":"9aec18ee3862d442d3046013ef21c6389a65c32caf8c8a0b8553da28926150b0"}
Sep 11 12:21:01.496989 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.496938    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67fcb57878-lbbjf" podStartSLOduration=2.496919559 podStartE2EDuration="2.496919559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:21:01.495415216 +0000 UTC m=+393.058194982" watchObservedRunningTime="2026-09-11 12:21:01.496919559 +0000 UTC m=+393.059699322"
Sep 11 12:21:01.566966 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.566933    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"]
Sep 11 12:21:01.571051 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.571034    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:01.575009 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.574988    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 11 12:21:01.575125 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.575110    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 11 12:21:01.575415 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.575395    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-w7rrr\""
Sep 11 12:21:01.575463 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.575421    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 12:21:01.575782 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.575769    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 11 12:21:01.595320 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.595298    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"]
Sep 11 12:21:01.618810 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.618787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a962a508-a4e4-4e45-af98-e8c095cafc67-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-x7rj7\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:01.618909 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.618832    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a962a508-a4e4-4e45-af98-e8c095cafc67-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-x7rj7\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:01.618909 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.618865    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a962a508-a4e4-4e45-af98-e8c095cafc67-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-x7rj7\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:01.618976 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.618924    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxq9q\" (UniqueName: \"kubernetes.io/projected/a962a508-a4e4-4e45-af98-e8c095cafc67-kube-api-access-kxq9q\") pod \"istiod-openshift-gateway-7b7995d5cd-x7rj7\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:01.618976 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.618966    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a962a508-a4e4-4e45-af98-e8c095cafc67-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-x7rj7\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:01.619048 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.619029    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a962a508-a4e4-4e45-af98-e8c095cafc67-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-x7rj7\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:01.619080 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.619067    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a962a508-a4e4-4e45-af98-e8c095cafc67-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-x7rj7\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:01.719502 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.719424    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a962a508-a4e4-4e45-af98-e8c095cafc67-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-x7rj7\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:01.719502 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.719463    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxq9q\" (UniqueName: \"kubernetes.io/projected/a962a508-a4e4-4e45-af98-e8c095cafc67-kube-api-access-kxq9q\") pod \"istiod-openshift-gateway-7b7995d5cd-x7rj7\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:01.719502 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.719489    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a962a508-a4e4-4e45-af98-e8c095cafc67-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-x7rj7\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:01.719727 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.719524    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a962a508-a4e4-4e45-af98-e8c095cafc67-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-x7rj7\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:01.719727 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.719550    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a962a508-a4e4-4e45-af98-e8c095cafc67-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-x7rj7\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:01.719727 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.719568    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a962a508-a4e4-4e45-af98-e8c095cafc67-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-x7rj7\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:01.719727 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.719602    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a962a508-a4e4-4e45-af98-e8c095cafc67-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-x7rj7\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:01.720089 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.720054    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a962a508-a4e4-4e45-af98-e8c095cafc67-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-x7rj7\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:01.722069 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.722040    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a962a508-a4e4-4e45-af98-e8c095cafc67-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-x7rj7\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:01.722185 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.722069    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a962a508-a4e4-4e45-af98-e8c095cafc67-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-x7rj7\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:01.722285 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.722207    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a962a508-a4e4-4e45-af98-e8c095cafc67-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-x7rj7\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:01.722340 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.722296    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a962a508-a4e4-4e45-af98-e8c095cafc67-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-x7rj7\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:01.732105 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.732083    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a962a508-a4e4-4e45-af98-e8c095cafc67-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-x7rj7\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:01.735108 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.735083    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxq9q\" (UniqueName: \"kubernetes.io/projected/a962a508-a4e4-4e45-af98-e8c095cafc67-kube-api-access-kxq9q\") pod \"istiod-openshift-gateway-7b7995d5cd-x7rj7\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:01.880044 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:01.880002    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:02.179162 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:02.179108    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"]
Sep 11 12:21:02.183935 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:21:02.183903    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda962a508_a4e4_4e45_af98_e8c095cafc67.slice/crio-0aff852afc546d7e44db08395789998af11c459d2ff35cdc7cf824b2f8151b76 WatchSource:0}: Error finding container 0aff852afc546d7e44db08395789998af11c459d2ff35cdc7cf824b2f8151b76: Status 404 returned error can't find the container with id 0aff852afc546d7e44db08395789998af11c459d2ff35cdc7cf824b2f8151b76
Sep 11 12:21:02.476611 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:02.476506    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7" event={"ID":"a962a508-a4e4-4e45-af98-e8c095cafc67","Type":"ContainerStarted","Data":"0aff852afc546d7e44db08395789998af11c459d2ff35cdc7cf824b2f8151b76"}
Sep 11 12:21:04.487063 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:04.487020    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-7cjqr" event={"ID":"f25fe8ac-8170-4f84-a77c-8f077fe8b458","Type":"ContainerStarted","Data":"f4829f762dd6d2129ab6d0cdfe348e6152f23e677ca36244db2ebe0dfa3a9d52"}
Sep 11 12:21:04.529283 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:04.529229    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-7cjqr" podStartSLOduration=2.725074482 podStartE2EDuration="5.529214381s" totalImagesPullingTime="2.804139899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:21:04.528283266 +0000 UTC m=+396.091063029" watchObservedRunningTime="2026-09-11 12:21:04.529214381 +0000 UTC m=+396.091994143"
Sep 11 12:21:05.818489 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:05.818450    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 12:21:05.818774 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:05.818518    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 12:21:06.497476 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:06.497425    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7" event={"ID":"a962a508-a4e4-4e45-af98-e8c095cafc67","Type":"ContainerStarted","Data":"87d0eab2acc812838e8b11715bccbedd4a2dd632b35861c65131c35fb1f09b8a"}
Sep 11 12:21:06.497647 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:06.497613    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:06.499165 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:06.499128    2561 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-x7rj7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 12:21:06.499292 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:06.499202    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7" podUID="a962a508-a4e4-4e45-af98-e8c095cafc67" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 12:21:06.558964 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:06.558916    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7" podStartSLOduration=1.927552981 podStartE2EDuration="5.558901646s" totalImagesPullingTime="3.631348665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:21:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:21:06.556259045 +0000 UTC m=+398.119038808" watchObservedRunningTime="2026-09-11 12:21:06.558901646 +0000 UTC m=+398.121681410"
Sep 11 12:21:07.501905 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:07.501878    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:10.223032 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:10.222991    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:10.223507 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:10.223058    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:10.227908 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:10.227879    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:10.517746 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:10.517720    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67fcb57878-lbbjf"
Sep 11 12:21:10.570878 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:10.570842    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86589f6569-rmd2g"]
Sep 11 12:21:18.705289 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:18.705254    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-69c967fbf8-sh9sv"]
Sep 11 12:21:18.836058 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:18.836023    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-69c967fbf8-sh9sv"]
Sep 11 12:21:18.836214 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:18.836128    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-sh9sv"
Sep 11 12:21:18.839782 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:18.839591    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 12:21:18.841862 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:18.841821    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 12:21:18.842001 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:18.841889    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 12:21:18.842001 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:18.841967    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 12:21:18.842001 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:18.841906    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-7gndt\""
Sep 11 12:21:18.842233 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:18.842211    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 12:21:18.959690 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:18.959622    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw9bj\" (UniqueName: \"kubernetes.io/projected/a253c393-34c4-4d8a-8179-8474b2af390c-kube-api-access-kw9bj\") pod \"lws-controller-manager-69c967fbf8-sh9sv\" (UID: \"a253c393-34c4-4d8a-8179-8474b2af390c\") " pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-sh9sv"
Sep 11 12:21:18.959690 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:18.959662    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a253c393-34c4-4d8a-8179-8474b2af390c-manager-config\") pod \"lws-controller-manager-69c967fbf8-sh9sv\" (UID: \"a253c393-34c4-4d8a-8179-8474b2af390c\") " pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-sh9sv"
Sep 11 12:21:18.959690 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:18.959683    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a253c393-34c4-4d8a-8179-8474b2af390c-metrics-cert\") pod \"lws-controller-manager-69c967fbf8-sh9sv\" (UID: \"a253c393-34c4-4d8a-8179-8474b2af390c\") " pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-sh9sv"
Sep 11 12:21:18.959895 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:18.959749    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a253c393-34c4-4d8a-8179-8474b2af390c-cert\") pod \"lws-controller-manager-69c967fbf8-sh9sv\" (UID: \"a253c393-34c4-4d8a-8179-8474b2af390c\") " pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-sh9sv"
Sep 11 12:21:19.060969 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:19.060935    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw9bj\" (UniqueName: \"kubernetes.io/projected/a253c393-34c4-4d8a-8179-8474b2af390c-kube-api-access-kw9bj\") pod \"lws-controller-manager-69c967fbf8-sh9sv\" (UID: \"a253c393-34c4-4d8a-8179-8474b2af390c\") " pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-sh9sv"
Sep 11 12:21:19.061121 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:19.060981    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a253c393-34c4-4d8a-8179-8474b2af390c-manager-config\") pod \"lws-controller-manager-69c967fbf8-sh9sv\" (UID: \"a253c393-34c4-4d8a-8179-8474b2af390c\") " pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-sh9sv"
Sep 11 12:21:19.061121 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:19.061009    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a253c393-34c4-4d8a-8179-8474b2af390c-metrics-cert\") pod \"lws-controller-manager-69c967fbf8-sh9sv\" (UID: \"a253c393-34c4-4d8a-8179-8474b2af390c\") " pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-sh9sv"
Sep 11 12:21:19.061121 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:19.061039    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a253c393-34c4-4d8a-8179-8474b2af390c-cert\") pod \"lws-controller-manager-69c967fbf8-sh9sv\" (UID: \"a253c393-34c4-4d8a-8179-8474b2af390c\") " pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-sh9sv"
Sep 11 12:21:19.061726 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:19.061704    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a253c393-34c4-4d8a-8179-8474b2af390c-manager-config\") pod \"lws-controller-manager-69c967fbf8-sh9sv\" (UID: \"a253c393-34c4-4d8a-8179-8474b2af390c\") " pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-sh9sv"
Sep 11 12:21:19.063675 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:19.063652    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a253c393-34c4-4d8a-8179-8474b2af390c-cert\") pod \"lws-controller-manager-69c967fbf8-sh9sv\" (UID: \"a253c393-34c4-4d8a-8179-8474b2af390c\") " pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-sh9sv"
Sep 11 12:21:19.064061 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:19.064044    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a253c393-34c4-4d8a-8179-8474b2af390c-metrics-cert\") pod \"lws-controller-manager-69c967fbf8-sh9sv\" (UID: \"a253c393-34c4-4d8a-8179-8474b2af390c\") " pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-sh9sv"
Sep 11 12:21:19.073620 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:19.073592    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw9bj\" (UniqueName: \"kubernetes.io/projected/a253c393-34c4-4d8a-8179-8474b2af390c-kube-api-access-kw9bj\") pod \"lws-controller-manager-69c967fbf8-sh9sv\" (UID: \"a253c393-34c4-4d8a-8179-8474b2af390c\") " pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-sh9sv"
Sep 11 12:21:19.148314 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:19.148288    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-sh9sv"
Sep 11 12:21:19.464243 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:19.464215    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-69c967fbf8-sh9sv"]
Sep 11 12:21:19.464881 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:21:19.464856    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda253c393_34c4_4d8a_8179_8474b2af390c.slice/crio-da4d1417a3ec5f5f886763d4c65f1d8102c2f7c73f95bea82494babf4efa72b3 WatchSource:0}: Error finding container da4d1417a3ec5f5f886763d4c65f1d8102c2f7c73f95bea82494babf4efa72b3: Status 404 returned error can't find the container with id da4d1417a3ec5f5f886763d4c65f1d8102c2f7c73f95bea82494babf4efa72b3
Sep 11 12:21:19.551931 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:19.551903    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-sh9sv" event={"ID":"a253c393-34c4-4d8a-8179-8474b2af390c","Type":"ContainerStarted","Data":"da4d1417a3ec5f5f886763d4c65f1d8102c2f7c73f95bea82494babf4efa72b3"}
Sep 11 12:21:22.567261 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:22.567221    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-sh9sv" event={"ID":"a253c393-34c4-4d8a-8179-8474b2af390c","Type":"ContainerStarted","Data":"9af04d4e16530f4f8683400629c70136f1236cbadaa82ec172b2dbb0fa80df32"}
Sep 11 12:21:22.567620 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:22.567364    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-sh9sv"
Sep 11 12:21:22.618354 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:22.618302    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-sh9sv" podStartSLOduration=2.199989601 podStartE2EDuration="4.61828821s" totalImagesPullingTime="2.418298609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:21:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:21:22.610328234 +0000 UTC m=+414.173108008" watchObservedRunningTime="2026-09-11 12:21:22.61828821 +0000 UTC m=+414.181067972"
Sep 11 12:21:23.086236 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.086184    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"]
Sep 11 12:21:23.086450 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.086427    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7" podUID="a962a508-a4e4-4e45-af98-e8c095cafc67" containerName="discovery" containerID="cri-o://87d0eab2acc812838e8b11715bccbedd4a2dd632b35861c65131c35fb1f09b8a" gracePeriod=30
Sep 11 12:21:23.319634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.319608    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:23.398886 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.398813    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a962a508-a4e4-4e45-af98-e8c095cafc67-istio-csr-dns-cert\") pod \"a962a508-a4e4-4e45-af98-e8c095cafc67\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") "
Sep 11 12:21:23.398886 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.398875    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a962a508-a4e4-4e45-af98-e8c095cafc67-istio-token\") pod \"a962a508-a4e4-4e45-af98-e8c095cafc67\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") "
Sep 11 12:21:23.399045 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.398903    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a962a508-a4e4-4e45-af98-e8c095cafc67-local-certs\") pod \"a962a508-a4e4-4e45-af98-e8c095cafc67\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") "
Sep 11 12:21:23.399045 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.398948    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a962a508-a4e4-4e45-af98-e8c095cafc67-istio-kubeconfig\") pod \"a962a508-a4e4-4e45-af98-e8c095cafc67\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") "
Sep 11 12:21:23.399045 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.399007    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a962a508-a4e4-4e45-af98-e8c095cafc67-istio-csr-ca-configmap\") pod \"a962a508-a4e4-4e45-af98-e8c095cafc67\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") "
Sep 11 12:21:23.399045 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.399029    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a962a508-a4e4-4e45-af98-e8c095cafc67-cacerts\") pod \"a962a508-a4e4-4e45-af98-e8c095cafc67\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") "
Sep 11 12:21:23.399263 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.399055    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxq9q\" (UniqueName: \"kubernetes.io/projected/a962a508-a4e4-4e45-af98-e8c095cafc67-kube-api-access-kxq9q\") pod \"a962a508-a4e4-4e45-af98-e8c095cafc67\" (UID: \"a962a508-a4e4-4e45-af98-e8c095cafc67\") "
Sep 11 12:21:23.399461 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.399434    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a962a508-a4e4-4e45-af98-e8c095cafc67-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "a962a508-a4e4-4e45-af98-e8c095cafc67" (UID: "a962a508-a4e4-4e45-af98-e8c095cafc67"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:21:23.401690 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.401660    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a962a508-a4e4-4e45-af98-e8c095cafc67-cacerts" (OuterVolumeSpecName: "cacerts") pod "a962a508-a4e4-4e45-af98-e8c095cafc67" (UID: "a962a508-a4e4-4e45-af98-e8c095cafc67"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:21:23.401793 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.401683    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a962a508-a4e4-4e45-af98-e8c095cafc67-istio-token" (OuterVolumeSpecName: "istio-token") pod "a962a508-a4e4-4e45-af98-e8c095cafc67" (UID: "a962a508-a4e4-4e45-af98-e8c095cafc67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:21:23.401854 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.401802    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a962a508-a4e4-4e45-af98-e8c095cafc67-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "a962a508-a4e4-4e45-af98-e8c095cafc67" (UID: "a962a508-a4e4-4e45-af98-e8c095cafc67"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:21:23.402042 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.402007    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a962a508-a4e4-4e45-af98-e8c095cafc67-kube-api-access-kxq9q" (OuterVolumeSpecName: "kube-api-access-kxq9q") pod "a962a508-a4e4-4e45-af98-e8c095cafc67" (UID: "a962a508-a4e4-4e45-af98-e8c095cafc67"). InnerVolumeSpecName "kube-api-access-kxq9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:21:23.402150 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.402055    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a962a508-a4e4-4e45-af98-e8c095cafc67-local-certs" (OuterVolumeSpecName: "local-certs") pod "a962a508-a4e4-4e45-af98-e8c095cafc67" (UID: "a962a508-a4e4-4e45-af98-e8c095cafc67"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:21:23.402150 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.402095    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a962a508-a4e4-4e45-af98-e8c095cafc67-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "a962a508-a4e4-4e45-af98-e8c095cafc67" (UID: "a962a508-a4e4-4e45-af98-e8c095cafc67"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:21:23.500010 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.499973    2561 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a962a508-a4e4-4e45-af98-e8c095cafc67-local-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:21:23.500010 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.500005    2561 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a962a508-a4e4-4e45-af98-e8c095cafc67-istio-kubeconfig\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:21:23.500010 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.500016    2561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a962a508-a4e4-4e45-af98-e8c095cafc67-istio-csr-ca-configmap\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:21:23.500271 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.500028    2561 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a962a508-a4e4-4e45-af98-e8c095cafc67-cacerts\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:21:23.500271 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.500040    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxq9q\" (UniqueName: \"kubernetes.io/projected/a962a508-a4e4-4e45-af98-e8c095cafc67-kube-api-access-kxq9q\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:21:23.500271 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.500054    2561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a962a508-a4e4-4e45-af98-e8c095cafc67-istio-csr-dns-cert\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:21:23.500271 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.500064    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a962a508-a4e4-4e45-af98-e8c095cafc67-istio-token\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:21:23.572878 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.572845    2561 generic.go:356] "Generic (PLEG): container finished" podID="a962a508-a4e4-4e45-af98-e8c095cafc67" containerID="87d0eab2acc812838e8b11715bccbedd4a2dd632b35861c65131c35fb1f09b8a" exitCode=0
Sep 11 12:21:23.573304 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.572890    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"
Sep 11 12:21:23.573304 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.572923    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7" event={"ID":"a962a508-a4e4-4e45-af98-e8c095cafc67","Type":"ContainerDied","Data":"87d0eab2acc812838e8b11715bccbedd4a2dd632b35861c65131c35fb1f09b8a"}
Sep 11 12:21:23.573304 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.572969    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7" event={"ID":"a962a508-a4e4-4e45-af98-e8c095cafc67","Type":"ContainerDied","Data":"0aff852afc546d7e44db08395789998af11c459d2ff35cdc7cf824b2f8151b76"}
Sep 11 12:21:23.573304 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.572988    2561 scope.go:117] "RemoveContainer" containerID="87d0eab2acc812838e8b11715bccbedd4a2dd632b35861c65131c35fb1f09b8a"
Sep 11 12:21:23.581719 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.581702    2561 scope.go:117] "RemoveContainer" containerID="87d0eab2acc812838e8b11715bccbedd4a2dd632b35861c65131c35fb1f09b8a"
Sep 11 12:21:23.581958 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:21:23.581940    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d0eab2acc812838e8b11715bccbedd4a2dd632b35861c65131c35fb1f09b8a\": container with ID starting with 87d0eab2acc812838e8b11715bccbedd4a2dd632b35861c65131c35fb1f09b8a not found: ID does not exist" containerID="87d0eab2acc812838e8b11715bccbedd4a2dd632b35861c65131c35fb1f09b8a"
Sep 11 12:21:23.582023 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.581966    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d0eab2acc812838e8b11715bccbedd4a2dd632b35861c65131c35fb1f09b8a"} err="failed to get container status \"87d0eab2acc812838e8b11715bccbedd4a2dd632b35861c65131c35fb1f09b8a\": rpc error: code = NotFound desc = could not find container \"87d0eab2acc812838e8b11715bccbedd4a2dd632b35861c65131c35fb1f09b8a\": container with ID starting with 87d0eab2acc812838e8b11715bccbedd4a2dd632b35861c65131c35fb1f09b8a not found: ID does not exist"
Sep 11 12:21:23.617720 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.617699    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"]
Sep 11 12:21:23.625648 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:23.625627    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x7rj7"]
Sep 11 12:21:24.932372 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:24.932343    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a962a508-a4e4-4e45-af98-e8c095cafc67" path="/var/lib/kubelet/pods/a962a508-a4e4-4e45-af98-e8c095cafc67/volumes"
Sep 11 12:21:33.575812 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:33.575777    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-69c967fbf8-sh9sv"
Sep 11 12:21:35.589401 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:35.589353    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86589f6569-rmd2g" podUID="92fe1048-0690-4d1e-bced-15ea06898832" containerName="console" containerID="cri-o://2b6ad77ad07ec57ee29082f838be1cd553c643d09a977b5af173ae6a27052965" gracePeriod=15
Sep 11 12:21:35.824846 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:35.824826    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:21:35.910962 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:35.910885    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92fe1048-0690-4d1e-bced-15ea06898832-trusted-ca-bundle\") pod \"92fe1048-0690-4d1e-bced-15ea06898832\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") "
Sep 11 12:21:35.910962 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:35.910927    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92fe1048-0690-4d1e-bced-15ea06898832-service-ca\") pod \"92fe1048-0690-4d1e-bced-15ea06898832\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") "
Sep 11 12:21:35.911168 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:35.911063    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92fe1048-0690-4d1e-bced-15ea06898832-oauth-serving-cert\") pod \"92fe1048-0690-4d1e-bced-15ea06898832\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") "
Sep 11 12:21:35.911168 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:35.911100    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92fe1048-0690-4d1e-bced-15ea06898832-console-oauth-config\") pod \"92fe1048-0690-4d1e-bced-15ea06898832\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") "
Sep 11 12:21:35.911168 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:35.911159    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92fe1048-0690-4d1e-bced-15ea06898832-console-serving-cert\") pod \"92fe1048-0690-4d1e-bced-15ea06898832\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") "
Sep 11 12:21:35.911339 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:35.911206    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6jv4\" (UniqueName: \"kubernetes.io/projected/92fe1048-0690-4d1e-bced-15ea06898832-kube-api-access-n6jv4\") pod \"92fe1048-0690-4d1e-bced-15ea06898832\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") "
Sep 11 12:21:35.911339 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:35.911238    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92fe1048-0690-4d1e-bced-15ea06898832-console-config\") pod \"92fe1048-0690-4d1e-bced-15ea06898832\" (UID: \"92fe1048-0690-4d1e-bced-15ea06898832\") "
Sep 11 12:21:35.911339 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:35.911305    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92fe1048-0690-4d1e-bced-15ea06898832-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "92fe1048-0690-4d1e-bced-15ea06898832" (UID: "92fe1048-0690-4d1e-bced-15ea06898832"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:21:35.911482 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:35.911340    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92fe1048-0690-4d1e-bced-15ea06898832-service-ca" (OuterVolumeSpecName: "service-ca") pod "92fe1048-0690-4d1e-bced-15ea06898832" (UID: "92fe1048-0690-4d1e-bced-15ea06898832"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:21:35.911482 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:35.911417    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92fe1048-0690-4d1e-bced-15ea06898832-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "92fe1048-0690-4d1e-bced-15ea06898832" (UID: "92fe1048-0690-4d1e-bced-15ea06898832"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:21:35.911658 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:35.911636    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92fe1048-0690-4d1e-bced-15ea06898832-trusted-ca-bundle\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:21:35.911719 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:35.911661    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92fe1048-0690-4d1e-bced-15ea06898832-service-ca\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:21:35.911719 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:35.911671    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92fe1048-0690-4d1e-bced-15ea06898832-oauth-serving-cert\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:21:35.911719 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:35.911688    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92fe1048-0690-4d1e-bced-15ea06898832-console-config" (OuterVolumeSpecName: "console-config") pod "92fe1048-0690-4d1e-bced-15ea06898832" (UID: "92fe1048-0690-4d1e-bced-15ea06898832"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:21:35.913307 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:35.913273    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92fe1048-0690-4d1e-bced-15ea06898832-kube-api-access-n6jv4" (OuterVolumeSpecName: "kube-api-access-n6jv4") pod "92fe1048-0690-4d1e-bced-15ea06898832" (UID: "92fe1048-0690-4d1e-bced-15ea06898832"). InnerVolumeSpecName "kube-api-access-n6jv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:21:35.913426 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:35.913343    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92fe1048-0690-4d1e-bced-15ea06898832-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "92fe1048-0690-4d1e-bced-15ea06898832" (UID: "92fe1048-0690-4d1e-bced-15ea06898832"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:21:35.913426 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:35.913363    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92fe1048-0690-4d1e-bced-15ea06898832-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "92fe1048-0690-4d1e-bced-15ea06898832" (UID: "92fe1048-0690-4d1e-bced-15ea06898832"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:21:36.012307 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:36.012275    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92fe1048-0690-4d1e-bced-15ea06898832-console-oauth-config\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:21:36.012307 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:36.012300    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92fe1048-0690-4d1e-bced-15ea06898832-console-serving-cert\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:21:36.012307 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:36.012309    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6jv4\" (UniqueName: \"kubernetes.io/projected/92fe1048-0690-4d1e-bced-15ea06898832-kube-api-access-n6jv4\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:21:36.012307 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:36.012317    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92fe1048-0690-4d1e-bced-15ea06898832-console-config\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:21:36.629031 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:36.628999    2561 generic.go:356] "Generic (PLEG): container finished" podID="92fe1048-0690-4d1e-bced-15ea06898832" containerID="2b6ad77ad07ec57ee29082f838be1cd553c643d09a977b5af173ae6a27052965" exitCode=0
Sep 11 12:21:36.629486 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:36.629046    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86589f6569-rmd2g"
Sep 11 12:21:36.629486 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:36.629069    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86589f6569-rmd2g" event={"ID":"92fe1048-0690-4d1e-bced-15ea06898832","Type":"ContainerDied","Data":"2b6ad77ad07ec57ee29082f838be1cd553c643d09a977b5af173ae6a27052965"}
Sep 11 12:21:36.629486 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:36.629108    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86589f6569-rmd2g" event={"ID":"92fe1048-0690-4d1e-bced-15ea06898832","Type":"ContainerDied","Data":"c7396559c2b1b6385e39acf195ab85c3f3ffa16e6b2550b6085bbbe3a2c90a74"}
Sep 11 12:21:36.629486 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:36.629126    2561 scope.go:117] "RemoveContainer" containerID="2b6ad77ad07ec57ee29082f838be1cd553c643d09a977b5af173ae6a27052965"
Sep 11 12:21:36.638283 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:36.638262    2561 scope.go:117] "RemoveContainer" containerID="2b6ad77ad07ec57ee29082f838be1cd553c643d09a977b5af173ae6a27052965"
Sep 11 12:21:36.638580 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:21:36.638557    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b6ad77ad07ec57ee29082f838be1cd553c643d09a977b5af173ae6a27052965\": container with ID starting with 2b6ad77ad07ec57ee29082f838be1cd553c643d09a977b5af173ae6a27052965 not found: ID does not exist" containerID="2b6ad77ad07ec57ee29082f838be1cd553c643d09a977b5af173ae6a27052965"
Sep 11 12:21:36.638646 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:36.638592    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b6ad77ad07ec57ee29082f838be1cd553c643d09a977b5af173ae6a27052965"} err="failed to get container status \"2b6ad77ad07ec57ee29082f838be1cd553c643d09a977b5af173ae6a27052965\": rpc error: code = NotFound desc = could not find container \"2b6ad77ad07ec57ee29082f838be1cd553c643d09a977b5af173ae6a27052965\": container with ID starting with 2b6ad77ad07ec57ee29082f838be1cd553c643d09a977b5af173ae6a27052965 not found: ID does not exist"
Sep 11 12:21:36.652511 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:36.652486    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86589f6569-rmd2g"]
Sep 11 12:21:36.656868 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:36.656846    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86589f6569-rmd2g"]
Sep 11 12:21:36.931033 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:21:36.930955    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92fe1048-0690-4d1e-bced-15ea06898832" path="/var/lib/kubelet/pods/92fe1048-0690-4d1e-bced-15ea06898832/volumes"
Sep 11 12:22:43.224722 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:43.224687    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-pf96w"]
Sep 11 12:22:43.225164 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:43.224993    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a962a508-a4e4-4e45-af98-e8c095cafc67" containerName="discovery"
Sep 11 12:22:43.225164 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:43.225007    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a962a508-a4e4-4e45-af98-e8c095cafc67" containerName="discovery"
Sep 11 12:22:43.225164 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:43.225032    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92fe1048-0690-4d1e-bced-15ea06898832" containerName="console"
Sep 11 12:22:43.225164 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:43.225037    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="92fe1048-0690-4d1e-bced-15ea06898832" containerName="console"
Sep 11 12:22:43.225164 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:43.225082    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="92fe1048-0690-4d1e-bced-15ea06898832" containerName="console"
Sep 11 12:22:43.225164 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:43.225094    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a962a508-a4e4-4e45-af98-e8c095cafc67" containerName="discovery"
Sep 11 12:22:43.229113 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:43.229097    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pf96w"
Sep 11 12:22:43.234079 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:43.234054    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 12:22:43.234255 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:43.234089    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 12:22:43.235283 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:43.235056    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 11 12:22:43.235283 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:43.235065    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 11 12:22:43.235396 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:43.235297    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tvpw6\""
Sep 11 12:22:43.247002 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:43.246980    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2p99\" (UniqueName: \"kubernetes.io/projected/2a372393-87e4-4660-a47b-5e2a0cefe67d-kube-api-access-b2p99\") pod \"kuadrant-console-plugin-fbdd4f445-pf96w\" (UID: \"2a372393-87e4-4660-a47b-5e2a0cefe67d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pf96w"
Sep 11 12:22:43.247094 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:43.247075    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a372393-87e4-4660-a47b-5e2a0cefe67d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-pf96w\" (UID: \"2a372393-87e4-4660-a47b-5e2a0cefe67d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pf96w"
Sep 11 12:22:43.247160 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:43.247124    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2a372393-87e4-4660-a47b-5e2a0cefe67d-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-pf96w\" (UID: \"2a372393-87e4-4660-a47b-5e2a0cefe67d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pf96w"
Sep 11 12:22:43.249317 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:43.249293    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-pf96w"]
Sep 11 12:22:43.347784 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:43.347751    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2p99\" (UniqueName: \"kubernetes.io/projected/2a372393-87e4-4660-a47b-5e2a0cefe67d-kube-api-access-b2p99\") pod \"kuadrant-console-plugin-fbdd4f445-pf96w\" (UID: \"2a372393-87e4-4660-a47b-5e2a0cefe67d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pf96w"
Sep 11 12:22:43.347932 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:43.347809    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a372393-87e4-4660-a47b-5e2a0cefe67d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-pf96w\" (UID: \"2a372393-87e4-4660-a47b-5e2a0cefe67d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pf96w"
Sep 11 12:22:43.347932 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:43.347837    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2a372393-87e4-4660-a47b-5e2a0cefe67d-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-pf96w\" (UID: \"2a372393-87e4-4660-a47b-5e2a0cefe67d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pf96w"
Sep 11 12:22:43.348043 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:22:43.347939    2561 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 11 12:22:43.348043 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:22:43.348006    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a372393-87e4-4660-a47b-5e2a0cefe67d-plugin-serving-cert podName:2a372393-87e4-4660-a47b-5e2a0cefe67d nodeName:}" failed. No retries permitted until 2026-09-11 12:22:43.847989898 +0000 UTC m=+495.410769639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/2a372393-87e4-4660-a47b-5e2a0cefe67d-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-pf96w" (UID: "2a372393-87e4-4660-a47b-5e2a0cefe67d") : secret "plugin-serving-cert" not found
Sep 11 12:22:43.348516 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:43.348494    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2a372393-87e4-4660-a47b-5e2a0cefe67d-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-pf96w\" (UID: \"2a372393-87e4-4660-a47b-5e2a0cefe67d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pf96w"
Sep 11 12:22:43.369526 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:43.369504    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2p99\" (UniqueName: \"kubernetes.io/projected/2a372393-87e4-4660-a47b-5e2a0cefe67d-kube-api-access-b2p99\") pod \"kuadrant-console-plugin-fbdd4f445-pf96w\" (UID: \"2a372393-87e4-4660-a47b-5e2a0cefe67d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pf96w"
Sep 11 12:22:43.851241 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:43.851175    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a372393-87e4-4660-a47b-5e2a0cefe67d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-pf96w\" (UID: \"2a372393-87e4-4660-a47b-5e2a0cefe67d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pf96w"
Sep 11 12:22:43.853604 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:43.853577    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a372393-87e4-4660-a47b-5e2a0cefe67d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-pf96w\" (UID: \"2a372393-87e4-4660-a47b-5e2a0cefe67d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pf96w"
Sep 11 12:22:44.138240 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:44.138149    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pf96w"
Sep 11 12:22:44.410546 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:44.410521    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-pf96w"]
Sep 11 12:22:44.412576 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:22:44.412548    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a372393_87e4_4660_a47b_5e2a0cefe67d.slice/crio-cf2cf3057f9f93cbf50d1339c696603f8c71c5db08c61070ac32aca3f32a6cef WatchSource:0}: Error finding container cf2cf3057f9f93cbf50d1339c696603f8c71c5db08c61070ac32aca3f32a6cef: Status 404 returned error can't find the container with id cf2cf3057f9f93cbf50d1339c696603f8c71c5db08c61070ac32aca3f32a6cef
Sep 11 12:22:44.902799 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:44.902763    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pf96w" event={"ID":"2a372393-87e4-4660-a47b-5e2a0cefe67d","Type":"ContainerStarted","Data":"cf2cf3057f9f93cbf50d1339c696603f8c71c5db08c61070ac32aca3f32a6cef"}
Sep 11 12:22:50.933651 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:50.933612    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pf96w" event={"ID":"2a372393-87e4-4660-a47b-5e2a0cefe67d","Type":"ContainerStarted","Data":"8412f33f11dffc828e8391bb1489db09daab53d157f1dab11ec8aa5f3b2574db"}
Sep 11 12:22:50.951201 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:22:50.951142    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-pf96w" podStartSLOduration=2.03337877 podStartE2EDuration="7.951129228s" totalImagesPullingTime="5.917750458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:22:50.950516998 +0000 UTC m=+502.513296763" watchObservedRunningTime="2026-09-11 12:22:50.951129228 +0000 UTC m=+502.513908990"
Sep 11 12:23:41.539001 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:41.538925    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:23:41.541266 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:41.541240    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"
Sep 11 12:23:41.543503 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:41.543482    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 12:23:41.555112 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:41.555089    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:23:41.588891 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:41.588858    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:23:41.714692 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:41.714656    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvcdk\" (UniqueName: \"kubernetes.io/projected/9408ef02-4f93-4e61-88c8-39ef23bf0761-kube-api-access-dvcdk\") pod \"limitador-limitador-84994f7ffc-2g9lc\" (UID: \"9408ef02-4f93-4e61-88c8-39ef23bf0761\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"
Sep 11 12:23:41.714843 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:41.714717    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9408ef02-4f93-4e61-88c8-39ef23bf0761-config-file\") pod \"limitador-limitador-84994f7ffc-2g9lc\" (UID: \"9408ef02-4f93-4e61-88c8-39ef23bf0761\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"
Sep 11 12:23:41.815449 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:41.815383    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvcdk\" (UniqueName: \"kubernetes.io/projected/9408ef02-4f93-4e61-88c8-39ef23bf0761-kube-api-access-dvcdk\") pod \"limitador-limitador-84994f7ffc-2g9lc\" (UID: \"9408ef02-4f93-4e61-88c8-39ef23bf0761\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"
Sep 11 12:23:41.815449 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:41.815435    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9408ef02-4f93-4e61-88c8-39ef23bf0761-config-file\") pod \"limitador-limitador-84994f7ffc-2g9lc\" (UID: \"9408ef02-4f93-4e61-88c8-39ef23bf0761\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"
Sep 11 12:23:41.816025 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:41.816007    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9408ef02-4f93-4e61-88c8-39ef23bf0761-config-file\") pod \"limitador-limitador-84994f7ffc-2g9lc\" (UID: \"9408ef02-4f93-4e61-88c8-39ef23bf0761\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"
Sep 11 12:23:41.824636 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:41.824612    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvcdk\" (UniqueName: \"kubernetes.io/projected/9408ef02-4f93-4e61-88c8-39ef23bf0761-kube-api-access-dvcdk\") pod \"limitador-limitador-84994f7ffc-2g9lc\" (UID: \"9408ef02-4f93-4e61-88c8-39ef23bf0761\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"
Sep 11 12:23:41.850518 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:41.850492    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"
Sep 11 12:23:41.938241 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:41.938208    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hkkcx"]
Sep 11 12:23:41.940469 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:41.940449    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hkkcx"
Sep 11 12:23:41.942850 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:41.942830    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bmjtk\""
Sep 11 12:23:41.948563 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:41.948542    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hkkcx"]
Sep 11 12:23:42.096096 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:42.096015    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-nr5tc"]
Sep 11 12:23:42.098951 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:42.098925    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-nr5tc"
Sep 11 12:23:42.108119 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:42.108096    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nr5tc"]
Sep 11 12:23:42.118762 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:42.118739    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b8r4\" (UniqueName: \"kubernetes.io/projected/83ee95bf-9752-4636-af66-ea3a3141b82e-kube-api-access-7b8r4\") pod \"authorino-7bcfd8f9cd-hkkcx\" (UID: \"83ee95bf-9752-4636-af66-ea3a3141b82e\") " pod="kuadrant-system/authorino-7bcfd8f9cd-hkkcx"
Sep 11 12:23:42.132075 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:42.132053    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:23:42.134812 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:23:42.134789    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9408ef02_4f93_4e61_88c8_39ef23bf0761.slice/crio-02f18ed4e3a64aafd70b832ae19d85978b63c51e01373bece5c5c19b1b9f3e7b WatchSource:0}: Error finding container 02f18ed4e3a64aafd70b832ae19d85978b63c51e01373bece5c5c19b1b9f3e7b: Status 404 returned error can't find the container with id 02f18ed4e3a64aafd70b832ae19d85978b63c51e01373bece5c5c19b1b9f3e7b
Sep 11 12:23:42.219225 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:42.219171    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b8r4\" (UniqueName: \"kubernetes.io/projected/83ee95bf-9752-4636-af66-ea3a3141b82e-kube-api-access-7b8r4\") pod \"authorino-7bcfd8f9cd-hkkcx\" (UID: \"83ee95bf-9752-4636-af66-ea3a3141b82e\") " pod="kuadrant-system/authorino-7bcfd8f9cd-hkkcx"
Sep 11 12:23:42.219376 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:42.219311    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsrzq\" (UniqueName: \"kubernetes.io/projected/3fa9ec4c-c21e-431d-b7a4-89764bb53a18-kube-api-access-qsrzq\") pod \"authorino-59c974bd48-nr5tc\" (UID: \"3fa9ec4c-c21e-431d-b7a4-89764bb53a18\") " pod="kuadrant-system/authorino-59c974bd48-nr5tc"
Sep 11 12:23:42.237822 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:42.237787    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b8r4\" (UniqueName: \"kubernetes.io/projected/83ee95bf-9752-4636-af66-ea3a3141b82e-kube-api-access-7b8r4\") pod \"authorino-7bcfd8f9cd-hkkcx\" (UID: \"83ee95bf-9752-4636-af66-ea3a3141b82e\") " pod="kuadrant-system/authorino-7bcfd8f9cd-hkkcx"
Sep 11 12:23:42.249419 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:42.249400    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hkkcx"
Sep 11 12:23:42.320700 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:42.320673    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsrzq\" (UniqueName: \"kubernetes.io/projected/3fa9ec4c-c21e-431d-b7a4-89764bb53a18-kube-api-access-qsrzq\") pod \"authorino-59c974bd48-nr5tc\" (UID: \"3fa9ec4c-c21e-431d-b7a4-89764bb53a18\") " pod="kuadrant-system/authorino-59c974bd48-nr5tc"
Sep 11 12:23:42.329834 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:42.329810    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsrzq\" (UniqueName: \"kubernetes.io/projected/3fa9ec4c-c21e-431d-b7a4-89764bb53a18-kube-api-access-qsrzq\") pod \"authorino-59c974bd48-nr5tc\" (UID: \"3fa9ec4c-c21e-431d-b7a4-89764bb53a18\") " pod="kuadrant-system/authorino-59c974bd48-nr5tc"
Sep 11 12:23:42.407762 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:42.407696    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-nr5tc"
Sep 11 12:23:42.514895 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:42.514864    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hkkcx"]
Sep 11 12:23:42.517540 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:23:42.517509    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83ee95bf_9752_4636_af66_ea3a3141b82e.slice/crio-5880839c2ee9878dbcd47766f6fb06b71e2e61c36994b68f6c0ec19a5842e63f WatchSource:0}: Error finding container 5880839c2ee9878dbcd47766f6fb06b71e2e61c36994b68f6c0ec19a5842e63f: Status 404 returned error can't find the container with id 5880839c2ee9878dbcd47766f6fb06b71e2e61c36994b68f6c0ec19a5842e63f
Sep 11 12:23:42.675055 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:42.675028    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nr5tc"]
Sep 11 12:23:42.677606 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:23:42.677579    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fa9ec4c_c21e_431d_b7a4_89764bb53a18.slice/crio-d56b9cfbc6a4c552abcb8290e63a92156761a163f847eae2fa6fe4405bf5b95b WatchSource:0}: Error finding container d56b9cfbc6a4c552abcb8290e63a92156761a163f847eae2fa6fe4405bf5b95b: Status 404 returned error can't find the container with id d56b9cfbc6a4c552abcb8290e63a92156761a163f847eae2fa6fe4405bf5b95b
Sep 11 12:23:43.142112 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:43.142073    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-2g9lc" event={"ID":"9408ef02-4f93-4e61-88c8-39ef23bf0761","Type":"ContainerStarted","Data":"02f18ed4e3a64aafd70b832ae19d85978b63c51e01373bece5c5c19b1b9f3e7b"}
Sep 11 12:23:43.143620 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:43.143593    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-nr5tc" event={"ID":"3fa9ec4c-c21e-431d-b7a4-89764bb53a18","Type":"ContainerStarted","Data":"d56b9cfbc6a4c552abcb8290e63a92156761a163f847eae2fa6fe4405bf5b95b"}
Sep 11 12:23:43.144965 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:43.144932    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hkkcx" event={"ID":"83ee95bf-9752-4636-af66-ea3a3141b82e","Type":"ContainerStarted","Data":"5880839c2ee9878dbcd47766f6fb06b71e2e61c36994b68f6c0ec19a5842e63f"}
Sep 11 12:23:45.156147 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:45.156056    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hkkcx" event={"ID":"83ee95bf-9752-4636-af66-ea3a3141b82e","Type":"ContainerStarted","Data":"64bea74d0763ea5495c3d5167921db9bde3b560efcb54a568e7f8ee96a5e31e7"}
Sep 11 12:23:45.157627 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:45.157599    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-2g9lc" event={"ID":"9408ef02-4f93-4e61-88c8-39ef23bf0761","Type":"ContainerStarted","Data":"6cb09274b1eb99018cb027202f0dd38d9d5c0e21caa14c93913e1cb4f8ec6542"}
Sep 11 12:23:45.157749 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:45.157732    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"
Sep 11 12:23:45.172266 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:45.172229    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-hkkcx" podStartSLOduration=1.953493907 podStartE2EDuration="4.172215523s" totalImagesPullingTime="2.218721616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:23:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:23:45.171329259 +0000 UTC m=+556.734109022" watchObservedRunningTime="2026-09-11 12:23:45.172215523 +0000 UTC m=+556.734995277"
Sep 11 12:23:45.191239 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:45.191172    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-2g9lc" podStartSLOduration=1.5944014819999999 podStartE2EDuration="4.191157618s" totalImagesPullingTime="2.596756136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:23:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:23:45.189824163 +0000 UTC m=+556.752603928" watchObservedRunningTime="2026-09-11 12:23:45.191157618 +0000 UTC m=+556.753937381"
Sep 11 12:23:46.163016 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:46.162929    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-nr5tc" event={"ID":"3fa9ec4c-c21e-431d-b7a4-89764bb53a18","Type":"ContainerStarted","Data":"9f441351b31cd8d9fe03764112b33d6b453ce6459966e2a7e3cb22021a83ab9f"}
Sep 11 12:23:46.181387 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:46.181341    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-nr5tc" podStartSLOduration=1.076917901 podStartE2EDuration="4.181325741s" totalImagesPullingTime="3.10440784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:23:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:23:46.180139877 +0000 UTC m=+557.742919645" watchObservedRunningTime="2026-09-11 12:23:46.181325741 +0000 UTC m=+557.744105503"
Sep 11 12:23:46.208227 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:46.208179    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hkkcx"]
Sep 11 12:23:47.167482 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:47.167442    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-hkkcx" podUID="83ee95bf-9752-4636-af66-ea3a3141b82e" containerName="authorino" containerID="cri-o://64bea74d0763ea5495c3d5167921db9bde3b560efcb54a568e7f8ee96a5e31e7" gracePeriod=30
Sep 11 12:23:47.406987 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:47.406963    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hkkcx"
Sep 11 12:23:47.563093 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:47.563067    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7b8r4\" (UniqueName: \"kubernetes.io/projected/83ee95bf-9752-4636-af66-ea3a3141b82e-kube-api-access-7b8r4\") pod \"83ee95bf-9752-4636-af66-ea3a3141b82e\" (UID: \"83ee95bf-9752-4636-af66-ea3a3141b82e\") "
Sep 11 12:23:47.565157 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:47.565126    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ee95bf-9752-4636-af66-ea3a3141b82e-kube-api-access-7b8r4" (OuterVolumeSpecName: "kube-api-access-7b8r4") pod "83ee95bf-9752-4636-af66-ea3a3141b82e" (UID: "83ee95bf-9752-4636-af66-ea3a3141b82e"). InnerVolumeSpecName "kube-api-access-7b8r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:23:47.664383 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:47.664361    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b8r4\" (UniqueName: \"kubernetes.io/projected/83ee95bf-9752-4636-af66-ea3a3141b82e-kube-api-access-7b8r4\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:23:48.172791 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:48.172757    2561 generic.go:356] "Generic (PLEG): container finished" podID="83ee95bf-9752-4636-af66-ea3a3141b82e" containerID="64bea74d0763ea5495c3d5167921db9bde3b560efcb54a568e7f8ee96a5e31e7" exitCode=0
Sep 11 12:23:48.173167 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:48.172794    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hkkcx"
Sep 11 12:23:48.173167 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:48.172805    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hkkcx" event={"ID":"83ee95bf-9752-4636-af66-ea3a3141b82e","Type":"ContainerDied","Data":"64bea74d0763ea5495c3d5167921db9bde3b560efcb54a568e7f8ee96a5e31e7"}
Sep 11 12:23:48.173167 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:48.172839    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hkkcx" event={"ID":"83ee95bf-9752-4636-af66-ea3a3141b82e","Type":"ContainerDied","Data":"5880839c2ee9878dbcd47766f6fb06b71e2e61c36994b68f6c0ec19a5842e63f"}
Sep 11 12:23:48.173167 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:48.172856    2561 scope.go:117] "RemoveContainer" containerID="64bea74d0763ea5495c3d5167921db9bde3b560efcb54a568e7f8ee96a5e31e7"
Sep 11 12:23:48.181628 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:48.181605    2561 scope.go:117] "RemoveContainer" containerID="64bea74d0763ea5495c3d5167921db9bde3b560efcb54a568e7f8ee96a5e31e7"
Sep 11 12:23:48.181892 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:23:48.181872    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64bea74d0763ea5495c3d5167921db9bde3b560efcb54a568e7f8ee96a5e31e7\": container with ID starting with 64bea74d0763ea5495c3d5167921db9bde3b560efcb54a568e7f8ee96a5e31e7 not found: ID does not exist" containerID="64bea74d0763ea5495c3d5167921db9bde3b560efcb54a568e7f8ee96a5e31e7"
Sep 11 12:23:48.181943 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:48.181900    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64bea74d0763ea5495c3d5167921db9bde3b560efcb54a568e7f8ee96a5e31e7"} err="failed to get container status \"64bea74d0763ea5495c3d5167921db9bde3b560efcb54a568e7f8ee96a5e31e7\": rpc error: code = NotFound desc = could not find container \"64bea74d0763ea5495c3d5167921db9bde3b560efcb54a568e7f8ee96a5e31e7\": container with ID starting with 64bea74d0763ea5495c3d5167921db9bde3b560efcb54a568e7f8ee96a5e31e7 not found: ID does not exist"
Sep 11 12:23:48.194003 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:48.193981    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hkkcx"]
Sep 11 12:23:48.198826 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:48.198796    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hkkcx"]
Sep 11 12:23:48.936241 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:48.936183    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83ee95bf-9752-4636-af66-ea3a3141b82e" path="/var/lib/kubelet/pods/83ee95bf-9752-4636-af66-ea3a3141b82e/volumes"
Sep 11 12:23:56.164023 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:23:56.163990    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"
Sep 11 12:24:13.192446 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:13.192413    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-flbpx"]
Sep 11 12:24:13.192888 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:13.192745    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83ee95bf-9752-4636-af66-ea3a3141b82e" containerName="authorino"
Sep 11 12:24:13.192888 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:13.192756    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="83ee95bf-9752-4636-af66-ea3a3141b82e" containerName="authorino"
Sep 11 12:24:13.192888 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:13.192820    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="83ee95bf-9752-4636-af66-ea3a3141b82e" containerName="authorino"
Sep 11 12:24:13.194621 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:13.194604    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-flbpx"
Sep 11 12:24:13.202774 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:13.202752    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-flbpx"]
Sep 11 12:24:13.276709 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:13.276680    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fnnt\" (UniqueName: \"kubernetes.io/projected/37ab373a-1183-4a83-b7e8-559c68ac9ef7-kube-api-access-8fnnt\") pod \"authorino-557f876f69-flbpx\" (UID: \"37ab373a-1183-4a83-b7e8-559c68ac9ef7\") " pod="kuadrant-system/authorino-557f876f69-flbpx"
Sep 11 12:24:13.377328 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:13.377302    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fnnt\" (UniqueName: \"kubernetes.io/projected/37ab373a-1183-4a83-b7e8-559c68ac9ef7-kube-api-access-8fnnt\") pod \"authorino-557f876f69-flbpx\" (UID: \"37ab373a-1183-4a83-b7e8-559c68ac9ef7\") " pod="kuadrant-system/authorino-557f876f69-flbpx"
Sep 11 12:24:13.398327 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:13.398306    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fnnt\" (UniqueName: \"kubernetes.io/projected/37ab373a-1183-4a83-b7e8-559c68ac9ef7-kube-api-access-8fnnt\") pod \"authorino-557f876f69-flbpx\" (UID: \"37ab373a-1183-4a83-b7e8-559c68ac9ef7\") " pod="kuadrant-system/authorino-557f876f69-flbpx"
Sep 11 12:24:13.436982 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:13.436954    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-flbpx"]
Sep 11 12:24:13.437158 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:13.437148    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-flbpx"
Sep 11 12:24:13.474456 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:13.474422    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8459c49c9d-lrq56"]
Sep 11 12:24:13.476829 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:13.476809    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8459c49c9d-lrq56"
Sep 11 12:24:13.479724 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:13.479701    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 12:24:13.487061 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:13.486553    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8459c49c9d-lrq56"]
Sep 11 12:24:13.579309 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:13.579279    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c31718ed-598b-489d-8560-8fc3c830bcf2-tls-cert\") pod \"authorino-8459c49c9d-lrq56\" (UID: \"c31718ed-598b-489d-8560-8fc3c830bcf2\") " pod="kuadrant-system/authorino-8459c49c9d-lrq56"
Sep 11 12:24:13.579467 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:13.579327    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rt8j\" (UniqueName: \"kubernetes.io/projected/c31718ed-598b-489d-8560-8fc3c830bcf2-kube-api-access-7rt8j\") pod \"authorino-8459c49c9d-lrq56\" (UID: \"c31718ed-598b-489d-8560-8fc3c830bcf2\") " pod="kuadrant-system/authorino-8459c49c9d-lrq56"
Sep 11 12:24:13.679762 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:13.679734    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c31718ed-598b-489d-8560-8fc3c830bcf2-tls-cert\") pod \"authorino-8459c49c9d-lrq56\" (UID: \"c31718ed-598b-489d-8560-8fc3c830bcf2\") " pod="kuadrant-system/authorino-8459c49c9d-lrq56"
Sep 11 12:24:13.679963 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:13.679773    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rt8j\" (UniqueName: \"kubernetes.io/projected/c31718ed-598b-489d-8560-8fc3c830bcf2-kube-api-access-7rt8j\") pod \"authorino-8459c49c9d-lrq56\" (UID: \"c31718ed-598b-489d-8560-8fc3c830bcf2\") " pod="kuadrant-system/authorino-8459c49c9d-lrq56"
Sep 11 12:24:13.682128 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:13.682102    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c31718ed-598b-489d-8560-8fc3c830bcf2-tls-cert\") pod \"authorino-8459c49c9d-lrq56\" (UID: \"c31718ed-598b-489d-8560-8fc3c830bcf2\") " pod="kuadrant-system/authorino-8459c49c9d-lrq56"
Sep 11 12:24:13.689093 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:13.689065    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rt8j\" (UniqueName: \"kubernetes.io/projected/c31718ed-598b-489d-8560-8fc3c830bcf2-kube-api-access-7rt8j\") pod \"authorino-8459c49c9d-lrq56\" (UID: \"c31718ed-598b-489d-8560-8fc3c830bcf2\") " pod="kuadrant-system/authorino-8459c49c9d-lrq56"
Sep 11 12:24:13.705744 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:13.705688    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-flbpx"]
Sep 11 12:24:13.708745 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:24:13.708722    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37ab373a_1183_4a83_b7e8_559c68ac9ef7.slice/crio-67b3ac7bf1170dc2ccf21373db44d99fe7006f8cef3e261c64746d43662336f9 WatchSource:0}: Error finding container 67b3ac7bf1170dc2ccf21373db44d99fe7006f8cef3e261c64746d43662336f9: Status 404 returned error can't find the container with id 67b3ac7bf1170dc2ccf21373db44d99fe7006f8cef3e261c64746d43662336f9
Sep 11 12:24:13.788552 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:13.788531    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8459c49c9d-lrq56"
Sep 11 12:24:14.054296 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:14.054134    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8459c49c9d-lrq56"]
Sep 11 12:24:14.056744 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:24:14.056719    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc31718ed_598b_489d_8560_8fc3c830bcf2.slice/crio-cb5c42ac4eda00d04f2f3204a3128d9d5096811375070dc0ff81c021e2330ff7 WatchSource:0}: Error finding container cb5c42ac4eda00d04f2f3204a3128d9d5096811375070dc0ff81c021e2330ff7: Status 404 returned error can't find the container with id cb5c42ac4eda00d04f2f3204a3128d9d5096811375070dc0ff81c021e2330ff7
Sep 11 12:24:14.282267 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:14.282224    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-flbpx" event={"ID":"37ab373a-1183-4a83-b7e8-559c68ac9ef7","Type":"ContainerStarted","Data":"67b3ac7bf1170dc2ccf21373db44d99fe7006f8cef3e261c64746d43662336f9"}
Sep 11 12:24:14.288903 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:14.288863    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8459c49c9d-lrq56" event={"ID":"c31718ed-598b-489d-8560-8fc3c830bcf2","Type":"ContainerStarted","Data":"cb5c42ac4eda00d04f2f3204a3128d9d5096811375070dc0ff81c021e2330ff7"}
Sep 11 12:24:16.298088 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:16.298009    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8459c49c9d-lrq56" event={"ID":"c31718ed-598b-489d-8560-8fc3c830bcf2","Type":"ContainerStarted","Data":"ff5e465ee5531bb478635df8976a7a2287f9bcb99aa07e76fb54b709e9b3a301"}
Sep 11 12:24:16.314849 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:16.314806    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8459c49c9d-lrq56" podStartSLOduration=1.352567345 podStartE2EDuration="3.314793959s" totalImagesPullingTime="1.962226614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:24:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:24:16.313541967 +0000 UTC m=+587.876321731" watchObservedRunningTime="2026-09-11 12:24:16.314793959 +0000 UTC m=+587.877573721"
Sep 11 12:24:16.346245 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:16.346218    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nr5tc"]
Sep 11 12:24:16.346402 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:16.346383    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-nr5tc" podUID="3fa9ec4c-c21e-431d-b7a4-89764bb53a18" containerName="authorino" containerID="cri-o://9f441351b31cd8d9fe03764112b33d6b453ce6459966e2a7e3cb22021a83ab9f" gracePeriod=30
Sep 11 12:24:16.584617 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:16.584592    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-nr5tc"
Sep 11 12:24:16.709458 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:16.709432    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsrzq\" (UniqueName: \"kubernetes.io/projected/3fa9ec4c-c21e-431d-b7a4-89764bb53a18-kube-api-access-qsrzq\") pod \"3fa9ec4c-c21e-431d-b7a4-89764bb53a18\" (UID: \"3fa9ec4c-c21e-431d-b7a4-89764bb53a18\") "
Sep 11 12:24:16.711421 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:16.711390    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa9ec4c-c21e-431d-b7a4-89764bb53a18-kube-api-access-qsrzq" (OuterVolumeSpecName: "kube-api-access-qsrzq") pod "3fa9ec4c-c21e-431d-b7a4-89764bb53a18" (UID: "3fa9ec4c-c21e-431d-b7a4-89764bb53a18"). InnerVolumeSpecName "kube-api-access-qsrzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:24:16.810240 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:16.810213    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsrzq\" (UniqueName: \"kubernetes.io/projected/3fa9ec4c-c21e-431d-b7a4-89764bb53a18-kube-api-access-qsrzq\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:24:17.303137 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:17.303098    2561 generic.go:356] "Generic (PLEG): container finished" podID="3fa9ec4c-c21e-431d-b7a4-89764bb53a18" containerID="9f441351b31cd8d9fe03764112b33d6b453ce6459966e2a7e3cb22021a83ab9f" exitCode=0
Sep 11 12:24:17.303137 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:17.303133    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-nr5tc"
Sep 11 12:24:17.303589 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:17.303140    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-nr5tc" event={"ID":"3fa9ec4c-c21e-431d-b7a4-89764bb53a18","Type":"ContainerDied","Data":"9f441351b31cd8d9fe03764112b33d6b453ce6459966e2a7e3cb22021a83ab9f"}
Sep 11 12:24:17.303589 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:17.303207    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-nr5tc" event={"ID":"3fa9ec4c-c21e-431d-b7a4-89764bb53a18","Type":"ContainerDied","Data":"d56b9cfbc6a4c552abcb8290e63a92156761a163f847eae2fa6fe4405bf5b95b"}
Sep 11 12:24:17.303589 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:17.303219    2561 scope.go:117] "RemoveContainer" containerID="9f441351b31cd8d9fe03764112b33d6b453ce6459966e2a7e3cb22021a83ab9f"
Sep 11 12:24:17.311439 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:17.311417    2561 scope.go:117] "RemoveContainer" containerID="9f441351b31cd8d9fe03764112b33d6b453ce6459966e2a7e3cb22021a83ab9f"
Sep 11 12:24:17.311723 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:24:17.311695    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f441351b31cd8d9fe03764112b33d6b453ce6459966e2a7e3cb22021a83ab9f\": container with ID starting with 9f441351b31cd8d9fe03764112b33d6b453ce6459966e2a7e3cb22021a83ab9f not found: ID does not exist" containerID="9f441351b31cd8d9fe03764112b33d6b453ce6459966e2a7e3cb22021a83ab9f"
Sep 11 12:24:17.311770 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:17.311733    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f441351b31cd8d9fe03764112b33d6b453ce6459966e2a7e3cb22021a83ab9f"} err="failed to get container status \"9f441351b31cd8d9fe03764112b33d6b453ce6459966e2a7e3cb22021a83ab9f\": rpc error: code = NotFound desc = could not find container \"9f441351b31cd8d9fe03764112b33d6b453ce6459966e2a7e3cb22021a83ab9f\": container with ID starting with 9f441351b31cd8d9fe03764112b33d6b453ce6459966e2a7e3cb22021a83ab9f not found: ID does not exist"
Sep 11 12:24:17.319264 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:17.319243    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nr5tc"]
Sep 11 12:24:17.323681 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:17.323662    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nr5tc"]
Sep 11 12:24:18.931298 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:18.931264    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fa9ec4c-c21e-431d-b7a4-89764bb53a18" path="/var/lib/kubelet/pods/3fa9ec4c-c21e-431d-b7a4-89764bb53a18/volumes"
Sep 11 12:24:25.339907 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:25.339870    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-flbpx" event={"ID":"37ab373a-1183-4a83-b7e8-559c68ac9ef7","Type":"ContainerStarted","Data":"ce771d0774b9aa8a66160472c8713b060aa53db6fa59fe6222dc897cc4c7c318"}
Sep 11 12:24:25.340302 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:25.339910    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-flbpx" podUID="37ab373a-1183-4a83-b7e8-559c68ac9ef7" containerName="authorino" containerID="cri-o://ce771d0774b9aa8a66160472c8713b060aa53db6fa59fe6222dc897cc4c7c318" gracePeriod=30
Sep 11 12:24:25.365334 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:25.365290    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-flbpx" podStartSLOduration=1.7520727740000002 podStartE2EDuration="12.365267555s" totalImagesPullingTime="10.613194781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:24:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:24:25.363748992 +0000 UTC m=+596.926528756" watchObservedRunningTime="2026-09-11 12:24:25.365267555 +0000 UTC m=+596.928047368"
Sep 11 12:24:25.581488 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:25.581463    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-flbpx"
Sep 11 12:24:25.685211 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:25.685111    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fnnt\" (UniqueName: \"kubernetes.io/projected/37ab373a-1183-4a83-b7e8-559c68ac9ef7-kube-api-access-8fnnt\") pod \"37ab373a-1183-4a83-b7e8-559c68ac9ef7\" (UID: \"37ab373a-1183-4a83-b7e8-559c68ac9ef7\") "
Sep 11 12:24:25.687123 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:25.687091    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ab373a-1183-4a83-b7e8-559c68ac9ef7-kube-api-access-8fnnt" (OuterVolumeSpecName: "kube-api-access-8fnnt") pod "37ab373a-1183-4a83-b7e8-559c68ac9ef7" (UID: "37ab373a-1183-4a83-b7e8-559c68ac9ef7"). InnerVolumeSpecName "kube-api-access-8fnnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:24:25.785965 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:25.785941    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fnnt\" (UniqueName: \"kubernetes.io/projected/37ab373a-1183-4a83-b7e8-559c68ac9ef7-kube-api-access-8fnnt\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:24:26.345686 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:26.345655    2561 generic.go:356] "Generic (PLEG): container finished" podID="37ab373a-1183-4a83-b7e8-559c68ac9ef7" containerID="ce771d0774b9aa8a66160472c8713b060aa53db6fa59fe6222dc897cc4c7c318" exitCode=0
Sep 11 12:24:26.346062 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:26.345700    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-flbpx"
Sep 11 12:24:26.346062 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:26.345742    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-flbpx" event={"ID":"37ab373a-1183-4a83-b7e8-559c68ac9ef7","Type":"ContainerDied","Data":"ce771d0774b9aa8a66160472c8713b060aa53db6fa59fe6222dc897cc4c7c318"}
Sep 11 12:24:26.346062 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:26.345785    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-flbpx" event={"ID":"37ab373a-1183-4a83-b7e8-559c68ac9ef7","Type":"ContainerDied","Data":"67b3ac7bf1170dc2ccf21373db44d99fe7006f8cef3e261c64746d43662336f9"}
Sep 11 12:24:26.346062 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:26.345806    2561 scope.go:117] "RemoveContainer" containerID="ce771d0774b9aa8a66160472c8713b060aa53db6fa59fe6222dc897cc4c7c318"
Sep 11 12:24:26.354331 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:26.354312    2561 scope.go:117] "RemoveContainer" containerID="ce771d0774b9aa8a66160472c8713b060aa53db6fa59fe6222dc897cc4c7c318"
Sep 11 12:24:26.354613 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:24:26.354594    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce771d0774b9aa8a66160472c8713b060aa53db6fa59fe6222dc897cc4c7c318\": container with ID starting with ce771d0774b9aa8a66160472c8713b060aa53db6fa59fe6222dc897cc4c7c318 not found: ID does not exist" containerID="ce771d0774b9aa8a66160472c8713b060aa53db6fa59fe6222dc897cc4c7c318"
Sep 11 12:24:26.354663 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:26.354623    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce771d0774b9aa8a66160472c8713b060aa53db6fa59fe6222dc897cc4c7c318"} err="failed to get container status \"ce771d0774b9aa8a66160472c8713b060aa53db6fa59fe6222dc897cc4c7c318\": rpc error: code = NotFound desc = could not find container \"ce771d0774b9aa8a66160472c8713b060aa53db6fa59fe6222dc897cc4c7c318\": container with ID starting with ce771d0774b9aa8a66160472c8713b060aa53db6fa59fe6222dc897cc4c7c318 not found: ID does not exist"
Sep 11 12:24:26.365802 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:26.365766    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-flbpx"]
Sep 11 12:24:26.368011 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:26.367989    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-flbpx"]
Sep 11 12:24:26.931475 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:26.931444    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37ab373a-1183-4a83-b7e8-559c68ac9ef7" path="/var/lib/kubelet/pods/37ab373a-1183-4a83-b7e8-559c68ac9ef7/volumes"
Sep 11 12:24:28.848472 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:28.848437    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcfm6_88402687-4e5b-411c-be7a-4cb955f90425/ovn-acl-logging/0.log"
Sep 11 12:24:28.848884 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:24:28.848651    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcfm6_88402687-4e5b-411c-be7a-4cb955f90425/ovn-acl-logging/0.log"
Sep 11 12:25:01.164614 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:01.164424    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:25:23.630476 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:23.630441    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:25:28.830986 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:28.830954    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:25:34.831412 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:34.831376    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:25:51.002098 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.002068    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"]
Sep 11 12:25:51.002467 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.002378    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fa9ec4c-c21e-431d-b7a4-89764bb53a18" containerName="authorino"
Sep 11 12:25:51.002467 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.002391    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fa9ec4c-c21e-431d-b7a4-89764bb53a18" containerName="authorino"
Sep 11 12:25:51.002467 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.002405    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37ab373a-1183-4a83-b7e8-559c68ac9ef7" containerName="authorino"
Sep 11 12:25:51.002467 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.002411    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="37ab373a-1183-4a83-b7e8-559c68ac9ef7" containerName="authorino"
Sep 11 12:25:51.002467 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.002468    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fa9ec4c-c21e-431d-b7a4-89764bb53a18" containerName="authorino"
Sep 11 12:25:51.002627 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.002477    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="37ab373a-1183-4a83-b7e8-559c68ac9ef7" containerName="authorino"
Sep 11 12:25:51.004573 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.004553    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"
Sep 11 12:25:51.007040 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.007019    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 12:25:51.008033 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.008013    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 12:25:51.008256 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.008239    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 11 12:25:51.008340 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.008267    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-r6tj8\""
Sep 11 12:25:51.014849 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.014827    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"]
Sep 11 12:25:51.078908 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.078877    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ab1431-d2e2-4cd5-b2b6-900ddf211117-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zrfn7\" (UID: \"c9ab1431-d2e2-4cd5-b2b6-900ddf211117\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"
Sep 11 12:25:51.079038 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.078917    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mxhl\" (UniqueName: \"kubernetes.io/projected/c9ab1431-d2e2-4cd5-b2b6-900ddf211117-kube-api-access-4mxhl\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zrfn7\" (UID: \"c9ab1431-d2e2-4cd5-b2b6-900ddf211117\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"
Sep 11 12:25:51.079038 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.078947    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ab1431-d2e2-4cd5-b2b6-900ddf211117-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zrfn7\" (UID: \"c9ab1431-d2e2-4cd5-b2b6-900ddf211117\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"
Sep 11 12:25:51.079038 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.078984    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ab1431-d2e2-4cd5-b2b6-900ddf211117-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zrfn7\" (UID: \"c9ab1431-d2e2-4cd5-b2b6-900ddf211117\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"
Sep 11 12:25:51.079038 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.079027    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ab1431-d2e2-4cd5-b2b6-900ddf211117-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zrfn7\" (UID: \"c9ab1431-d2e2-4cd5-b2b6-900ddf211117\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"
Sep 11 12:25:51.079211 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.079072    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ab1431-d2e2-4cd5-b2b6-900ddf211117-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zrfn7\" (UID: \"c9ab1431-d2e2-4cd5-b2b6-900ddf211117\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"
Sep 11 12:25:51.180321 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.180294    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ab1431-d2e2-4cd5-b2b6-900ddf211117-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zrfn7\" (UID: \"c9ab1431-d2e2-4cd5-b2b6-900ddf211117\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"
Sep 11 12:25:51.180410 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.180323    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ab1431-d2e2-4cd5-b2b6-900ddf211117-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zrfn7\" (UID: \"c9ab1431-d2e2-4cd5-b2b6-900ddf211117\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"
Sep 11 12:25:51.180410 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.180340    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ab1431-d2e2-4cd5-b2b6-900ddf211117-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zrfn7\" (UID: \"c9ab1431-d2e2-4cd5-b2b6-900ddf211117\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"
Sep 11 12:25:51.180410 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.180382    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ab1431-d2e2-4cd5-b2b6-900ddf211117-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zrfn7\" (UID: \"c9ab1431-d2e2-4cd5-b2b6-900ddf211117\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"
Sep 11 12:25:51.180410 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.180408    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ab1431-d2e2-4cd5-b2b6-900ddf211117-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zrfn7\" (UID: \"c9ab1431-d2e2-4cd5-b2b6-900ddf211117\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"
Sep 11 12:25:51.180556 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.180439    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mxhl\" (UniqueName: \"kubernetes.io/projected/c9ab1431-d2e2-4cd5-b2b6-900ddf211117-kube-api-access-4mxhl\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zrfn7\" (UID: \"c9ab1431-d2e2-4cd5-b2b6-900ddf211117\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"
Sep 11 12:25:51.180809 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.180781    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ab1431-d2e2-4cd5-b2b6-900ddf211117-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zrfn7\" (UID: \"c9ab1431-d2e2-4cd5-b2b6-900ddf211117\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"
Sep 11 12:25:51.180902 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.180821    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ab1431-d2e2-4cd5-b2b6-900ddf211117-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zrfn7\" (UID: \"c9ab1431-d2e2-4cd5-b2b6-900ddf211117\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"
Sep 11 12:25:51.180902 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.180835    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ab1431-d2e2-4cd5-b2b6-900ddf211117-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zrfn7\" (UID: \"c9ab1431-d2e2-4cd5-b2b6-900ddf211117\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"
Sep 11 12:25:51.182651 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.182626    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ab1431-d2e2-4cd5-b2b6-900ddf211117-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zrfn7\" (UID: \"c9ab1431-d2e2-4cd5-b2b6-900ddf211117\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"
Sep 11 12:25:51.182746 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.182726    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ab1431-d2e2-4cd5-b2b6-900ddf211117-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zrfn7\" (UID: \"c9ab1431-d2e2-4cd5-b2b6-900ddf211117\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"
Sep 11 12:25:51.188094 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.188073    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mxhl\" (UniqueName: \"kubernetes.io/projected/c9ab1431-d2e2-4cd5-b2b6-900ddf211117-kube-api-access-4mxhl\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zrfn7\" (UID: \"c9ab1431-d2e2-4cd5-b2b6-900ddf211117\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"
Sep 11 12:25:51.315098 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.315006    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"
Sep 11 12:25:51.587993 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.587970    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"]
Sep 11 12:25:51.590527 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:25:51.590499    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ab1431_d2e2_4cd5_b2b6_900ddf211117.slice/crio-0d8ef2bee56c262c7a72857da1d0a500568df0bddee119ca685eb28752293a24 WatchSource:0}: Error finding container 0d8ef2bee56c262c7a72857da1d0a500568df0bddee119ca685eb28752293a24: Status 404 returned error can't find the container with id 0d8ef2bee56c262c7a72857da1d0a500568df0bddee119ca685eb28752293a24
Sep 11 12:25:51.591874 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.591854    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 12:25:51.705866 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:51.705838    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7" event={"ID":"c9ab1431-d2e2-4cd5-b2b6-900ddf211117","Type":"ContainerStarted","Data":"0d8ef2bee56c262c7a72857da1d0a500568df0bddee119ca685eb28752293a24"}
Sep 11 12:25:52.129108 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:52.128401    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:25:52.711980 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:52.711940    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7" event={"ID":"c9ab1431-d2e2-4cd5-b2b6-900ddf211117","Type":"ContainerStarted","Data":"3a1e26e40e6f4f1de14dceeb8234c88e1afec2ed42c4084d2f4c225220055908"}
Sep 11 12:25:52.730285 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:52.730225    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7" podStartSLOduration=1.7466768799999999 podStartE2EDuration="2.730205133s" totalImagesPullingTime="983.528253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:25:52.729956047 +0000 UTC m=+684.292735813" watchObservedRunningTime="2026-09-11 12:25:52.730205133 +0000 UTC m=+684.292984894"
Sep 11 12:25:59.126628 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:25:59.126592    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:26:01.315822 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:01.315785    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"
Sep 11 12:26:01.316245 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:01.315851    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"
Sep 11 12:26:01.328494 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:01.328466    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"
Sep 11 12:26:01.761171 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:01.761141    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zrfn7"
Sep 11 12:26:06.031859 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:06.031822    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:26:20.601893 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:20.601857    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"]
Sep 11 12:26:20.604437 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:20.604416    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"
Sep 11 12:26:20.606697 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:20.606673    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 12:26:20.617049 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:20.617028    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"]
Sep 11 12:26:20.736989 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:20.736956    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b270b5fd-d777-4236-9c22-228a4fdfe291-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z\" (UID: \"b270b5fd-d777-4236-9c22-228a4fdfe291\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"
Sep 11 12:26:20.737122 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:20.736992    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b270b5fd-d777-4236-9c22-228a4fdfe291-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z\" (UID: \"b270b5fd-d777-4236-9c22-228a4fdfe291\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"
Sep 11 12:26:20.737122 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:20.737026    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b270b5fd-d777-4236-9c22-228a4fdfe291-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z\" (UID: \"b270b5fd-d777-4236-9c22-228a4fdfe291\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"
Sep 11 12:26:20.737122 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:20.737087    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b270b5fd-d777-4236-9c22-228a4fdfe291-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z\" (UID: \"b270b5fd-d777-4236-9c22-228a4fdfe291\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"
Sep 11 12:26:20.737122 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:20.737112    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2db6k\" (UniqueName: \"kubernetes.io/projected/b270b5fd-d777-4236-9c22-228a4fdfe291-kube-api-access-2db6k\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z\" (UID: \"b270b5fd-d777-4236-9c22-228a4fdfe291\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"
Sep 11 12:26:20.737305 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:20.737177    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b270b5fd-d777-4236-9c22-228a4fdfe291-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z\" (UID: \"b270b5fd-d777-4236-9c22-228a4fdfe291\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"
Sep 11 12:26:20.838442 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:20.838413    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b270b5fd-d777-4236-9c22-228a4fdfe291-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z\" (UID: \"b270b5fd-d777-4236-9c22-228a4fdfe291\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"
Sep 11 12:26:20.838553 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:20.838494    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b270b5fd-d777-4236-9c22-228a4fdfe291-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z\" (UID: \"b270b5fd-d777-4236-9c22-228a4fdfe291\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"
Sep 11 12:26:20.838553 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:20.838523    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b270b5fd-d777-4236-9c22-228a4fdfe291-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z\" (UID: \"b270b5fd-d777-4236-9c22-228a4fdfe291\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"
Sep 11 12:26:20.838623 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:20.838554    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b270b5fd-d777-4236-9c22-228a4fdfe291-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z\" (UID: \"b270b5fd-d777-4236-9c22-228a4fdfe291\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"
Sep 11 12:26:20.838623 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:20.838582    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b270b5fd-d777-4236-9c22-228a4fdfe291-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z\" (UID: \"b270b5fd-d777-4236-9c22-228a4fdfe291\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"
Sep 11 12:26:20.838765 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:20.838741    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2db6k\" (UniqueName: \"kubernetes.io/projected/b270b5fd-d777-4236-9c22-228a4fdfe291-kube-api-access-2db6k\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z\" (UID: \"b270b5fd-d777-4236-9c22-228a4fdfe291\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"
Sep 11 12:26:20.838896 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:20.838875    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b270b5fd-d777-4236-9c22-228a4fdfe291-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z\" (UID: \"b270b5fd-d777-4236-9c22-228a4fdfe291\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"
Sep 11 12:26:20.838949 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:20.838904    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b270b5fd-d777-4236-9c22-228a4fdfe291-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z\" (UID: \"b270b5fd-d777-4236-9c22-228a4fdfe291\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"
Sep 11 12:26:20.839016 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:20.838994    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b270b5fd-d777-4236-9c22-228a4fdfe291-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z\" (UID: \"b270b5fd-d777-4236-9c22-228a4fdfe291\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"
Sep 11 12:26:20.840823 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:20.840806    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b270b5fd-d777-4236-9c22-228a4fdfe291-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z\" (UID: \"b270b5fd-d777-4236-9c22-228a4fdfe291\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"
Sep 11 12:26:20.841082 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:20.841060    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b270b5fd-d777-4236-9c22-228a4fdfe291-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z\" (UID: \"b270b5fd-d777-4236-9c22-228a4fdfe291\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"
Sep 11 12:26:20.846844 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:20.846822    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2db6k\" (UniqueName: \"kubernetes.io/projected/b270b5fd-d777-4236-9c22-228a4fdfe291-kube-api-access-2db6k\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z\" (UID: \"b270b5fd-d777-4236-9c22-228a4fdfe291\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"
Sep 11 12:26:20.913453 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:20.913393    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"
Sep 11 12:26:21.182557 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:21.182532    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"]
Sep 11 12:26:21.184814 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:26:21.184786    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb270b5fd_d777_4236_9c22_228a4fdfe291.slice/crio-60c46cf21d4c7d81c567e8059fe0f3c09193afb44529d80a0a9fda752f5fea00 WatchSource:0}: Error finding container 60c46cf21d4c7d81c567e8059fe0f3c09193afb44529d80a0a9fda752f5fea00: Status 404 returned error can't find the container with id 60c46cf21d4c7d81c567e8059fe0f3c09193afb44529d80a0a9fda752f5fea00
Sep 11 12:26:21.841438 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:21.841399    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z" event={"ID":"b270b5fd-d777-4236-9c22-228a4fdfe291","Type":"ContainerStarted","Data":"b947e5b694a53fc837b62759a3d5c7c2270ed6ffd53201a8556f886544a0fd43"}
Sep 11 12:26:21.841438 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:21.841442    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z" event={"ID":"b270b5fd-d777-4236-9c22-228a4fdfe291","Type":"ContainerStarted","Data":"60c46cf21d4c7d81c567e8059fe0f3c09193afb44529d80a0a9fda752f5fea00"}
Sep 11 12:26:21.863268 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:21.863211    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z" podStartSLOduration=1.69121505 podStartE2EDuration="1.863185085s" totalImagesPullingTime="171.970035ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:26:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:26:21.861397338 +0000 UTC m=+713.424177094" watchObservedRunningTime="2026-09-11 12:26:21.863185085 +0000 UTC m=+713.425964842"
Sep 11 12:26:22.635357 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:22.635315    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:26:30.914023 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:30.913993    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"
Sep 11 12:26:30.914479 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:30.914039    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"
Sep 11 12:26:30.931552 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:30.931531    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"
Sep 11 12:26:31.901102 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:26:31.901074    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z"
Sep 11 12:27:37.290373 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:37.290337    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-9c68dcdc5-dz8c4"]
Sep 11 12:27:37.293739 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:37.293718    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9c68dcdc5-dz8c4"
Sep 11 12:27:37.301666 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:37.301644    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9c68dcdc5-dz8c4"]
Sep 11 12:27:37.383355 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:37.383327    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1ff9a731-1667-4747-9f10-351e5ac7f0df-tls-cert\") pod \"authorino-9c68dcdc5-dz8c4\" (UID: \"1ff9a731-1667-4747-9f10-351e5ac7f0df\") " pod="kuadrant-system/authorino-9c68dcdc5-dz8c4"
Sep 11 12:27:37.383532 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:37.383379    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvb4n\" (UniqueName: \"kubernetes.io/projected/1ff9a731-1667-4747-9f10-351e5ac7f0df-kube-api-access-fvb4n\") pod \"authorino-9c68dcdc5-dz8c4\" (UID: \"1ff9a731-1667-4747-9f10-351e5ac7f0df\") " pod="kuadrant-system/authorino-9c68dcdc5-dz8c4"
Sep 11 12:27:37.484317 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:37.484284    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1ff9a731-1667-4747-9f10-351e5ac7f0df-tls-cert\") pod \"authorino-9c68dcdc5-dz8c4\" (UID: \"1ff9a731-1667-4747-9f10-351e5ac7f0df\") " pod="kuadrant-system/authorino-9c68dcdc5-dz8c4"
Sep 11 12:27:37.484462 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:37.484347    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvb4n\" (UniqueName: \"kubernetes.io/projected/1ff9a731-1667-4747-9f10-351e5ac7f0df-kube-api-access-fvb4n\") pod \"authorino-9c68dcdc5-dz8c4\" (UID: \"1ff9a731-1667-4747-9f10-351e5ac7f0df\") " pod="kuadrant-system/authorino-9c68dcdc5-dz8c4"
Sep 11 12:27:37.486725 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:37.486703    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1ff9a731-1667-4747-9f10-351e5ac7f0df-tls-cert\") pod \"authorino-9c68dcdc5-dz8c4\" (UID: \"1ff9a731-1667-4747-9f10-351e5ac7f0df\") " pod="kuadrant-system/authorino-9c68dcdc5-dz8c4"
Sep 11 12:27:37.493167 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:37.493142    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvb4n\" (UniqueName: \"kubernetes.io/projected/1ff9a731-1667-4747-9f10-351e5ac7f0df-kube-api-access-fvb4n\") pod \"authorino-9c68dcdc5-dz8c4\" (UID: \"1ff9a731-1667-4747-9f10-351e5ac7f0df\") " pod="kuadrant-system/authorino-9c68dcdc5-dz8c4"
Sep 11 12:27:37.602903 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:37.602831    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9c68dcdc5-dz8c4"
Sep 11 12:27:37.870471 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:37.870446    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9c68dcdc5-dz8c4"]
Sep 11 12:27:37.873130 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:27:37.873103    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ff9a731_1667_4747_9f10_351e5ac7f0df.slice/crio-11d13cb47c9901b2a0aa03e096f2e44b5578081f6ccaa14f3d7ab5459dc55be0 WatchSource:0}: Error finding container 11d13cb47c9901b2a0aa03e096f2e44b5578081f6ccaa14f3d7ab5459dc55be0: Status 404 returned error can't find the container with id 11d13cb47c9901b2a0aa03e096f2e44b5578081f6ccaa14f3d7ab5459dc55be0
Sep 11 12:27:38.179257 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:38.179156    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9c68dcdc5-dz8c4" event={"ID":"1ff9a731-1667-4747-9f10-351e5ac7f0df","Type":"ContainerStarted","Data":"11d13cb47c9901b2a0aa03e096f2e44b5578081f6ccaa14f3d7ab5459dc55be0"}
Sep 11 12:27:39.185627 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:39.185583    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9c68dcdc5-dz8c4" event={"ID":"1ff9a731-1667-4747-9f10-351e5ac7f0df","Type":"ContainerStarted","Data":"367b3cb89ddff31e24064e491cd5db4f049bdd3e483d85a40bf6993c9d43847b"}
Sep 11 12:27:39.209006 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:39.208952    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-9c68dcdc5-dz8c4" podStartSLOduration=1.518774335 podStartE2EDuration="2.20893628s" totalImagesPullingTime="690.161945ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:27:39.208349066 +0000 UTC m=+790.771128830" watchObservedRunningTime="2026-09-11 12:27:39.20893628 +0000 UTC m=+790.771716043"
Sep 11 12:27:39.236756 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:39.236713    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8459c49c9d-lrq56"]
Sep 11 12:27:39.239087 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:39.239006    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-8459c49c9d-lrq56" podUID="c31718ed-598b-489d-8560-8fc3c830bcf2" containerName="authorino" containerID="cri-o://ff5e465ee5531bb478635df8976a7a2287f9bcb99aa07e76fb54b709e9b3a301" gracePeriod=30
Sep 11 12:27:39.491259 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:39.491235    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8459c49c9d-lrq56"
Sep 11 12:27:39.603290 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:39.603243    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c31718ed-598b-489d-8560-8fc3c830bcf2-tls-cert\") pod \"c31718ed-598b-489d-8560-8fc3c830bcf2\" (UID: \"c31718ed-598b-489d-8560-8fc3c830bcf2\") "
Sep 11 12:27:39.603463 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:39.603361    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rt8j\" (UniqueName: \"kubernetes.io/projected/c31718ed-598b-489d-8560-8fc3c830bcf2-kube-api-access-7rt8j\") pod \"c31718ed-598b-489d-8560-8fc3c830bcf2\" (UID: \"c31718ed-598b-489d-8560-8fc3c830bcf2\") "
Sep 11 12:27:39.606007 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:39.605978    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31718ed-598b-489d-8560-8fc3c830bcf2-kube-api-access-7rt8j" (OuterVolumeSpecName: "kube-api-access-7rt8j") pod "c31718ed-598b-489d-8560-8fc3c830bcf2" (UID: "c31718ed-598b-489d-8560-8fc3c830bcf2"). InnerVolumeSpecName "kube-api-access-7rt8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:27:39.613330 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:39.613300    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c31718ed-598b-489d-8560-8fc3c830bcf2-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "c31718ed-598b-489d-8560-8fc3c830bcf2" (UID: "c31718ed-598b-489d-8560-8fc3c830bcf2"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:27:39.704530 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:39.704495    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rt8j\" (UniqueName: \"kubernetes.io/projected/c31718ed-598b-489d-8560-8fc3c830bcf2-kube-api-access-7rt8j\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:27:39.704530 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:39.704528    2561 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c31718ed-598b-489d-8560-8fc3c830bcf2-tls-cert\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:27:40.191749 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:40.191709    2561 generic.go:356] "Generic (PLEG): container finished" podID="c31718ed-598b-489d-8560-8fc3c830bcf2" containerID="ff5e465ee5531bb478635df8976a7a2287f9bcb99aa07e76fb54b709e9b3a301" exitCode=0
Sep 11 12:27:40.191749 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:40.191751    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8459c49c9d-lrq56"
Sep 11 12:27:40.192304 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:40.191802    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8459c49c9d-lrq56" event={"ID":"c31718ed-598b-489d-8560-8fc3c830bcf2","Type":"ContainerDied","Data":"ff5e465ee5531bb478635df8976a7a2287f9bcb99aa07e76fb54b709e9b3a301"}
Sep 11 12:27:40.192304 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:40.191850    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8459c49c9d-lrq56" event={"ID":"c31718ed-598b-489d-8560-8fc3c830bcf2","Type":"ContainerDied","Data":"cb5c42ac4eda00d04f2f3204a3128d9d5096811375070dc0ff81c021e2330ff7"}
Sep 11 12:27:40.192304 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:40.191878    2561 scope.go:117] "RemoveContainer" containerID="ff5e465ee5531bb478635df8976a7a2287f9bcb99aa07e76fb54b709e9b3a301"
Sep 11 12:27:40.201922 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:40.201905    2561 scope.go:117] "RemoveContainer" containerID="ff5e465ee5531bb478635df8976a7a2287f9bcb99aa07e76fb54b709e9b3a301"
Sep 11 12:27:40.202211 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:27:40.202177    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff5e465ee5531bb478635df8976a7a2287f9bcb99aa07e76fb54b709e9b3a301\": container with ID starting with ff5e465ee5531bb478635df8976a7a2287f9bcb99aa07e76fb54b709e9b3a301 not found: ID does not exist" containerID="ff5e465ee5531bb478635df8976a7a2287f9bcb99aa07e76fb54b709e9b3a301"
Sep 11 12:27:40.202309 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:40.202216    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff5e465ee5531bb478635df8976a7a2287f9bcb99aa07e76fb54b709e9b3a301"} err="failed to get container status \"ff5e465ee5531bb478635df8976a7a2287f9bcb99aa07e76fb54b709e9b3a301\": rpc error: code = NotFound desc = could not find container \"ff5e465ee5531bb478635df8976a7a2287f9bcb99aa07e76fb54b709e9b3a301\": container with ID starting with ff5e465ee5531bb478635df8976a7a2287f9bcb99aa07e76fb54b709e9b3a301 not found: ID does not exist"
Sep 11 12:27:40.211301 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:40.211278    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8459c49c9d-lrq56"]
Sep 11 12:27:40.214553 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:40.214531    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8459c49c9d-lrq56"]
Sep 11 12:27:40.932236 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:40.932204    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c31718ed-598b-489d-8560-8fc3c830bcf2" path="/var/lib/kubelet/pods/c31718ed-598b-489d-8560-8fc3c830bcf2/volumes"
Sep 11 12:27:58.114624 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:58.114552    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7612-f3df31"]
Sep 11 12:27:58.115013 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:58.114892    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c31718ed-598b-489d-8560-8fc3c830bcf2" containerName="authorino"
Sep 11 12:27:58.115013 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:58.114905    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c31718ed-598b-489d-8560-8fc3c830bcf2" containerName="authorino"
Sep 11 12:27:58.115013 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:58.114972    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c31718ed-598b-489d-8560-8fc3c830bcf2" containerName="authorino"
Sep 11 12:27:58.117951 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:58.117934    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-f3df31"
Sep 11 12:27:58.120809 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:58.120793    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h59p7\""
Sep 11 12:27:58.125312 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:58.125282    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-f3df31"]
Sep 11 12:27:58.265526 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:58.265495    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxx4k\" (UniqueName: \"kubernetes.io/projected/2d6ab821-525f-41c1-8a94-c83c1b9147aa-kube-api-access-wxx4k\") pod \"test-curl-7612-f3df31\" (UID: \"2d6ab821-525f-41c1-8a94-c83c1b9147aa\") " pod="opendatahub/test-curl-7612-f3df31"
Sep 11 12:27:58.366895 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:58.366813    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxx4k\" (UniqueName: \"kubernetes.io/projected/2d6ab821-525f-41c1-8a94-c83c1b9147aa-kube-api-access-wxx4k\") pod \"test-curl-7612-f3df31\" (UID: \"2d6ab821-525f-41c1-8a94-c83c1b9147aa\") " pod="opendatahub/test-curl-7612-f3df31"
Sep 11 12:27:58.389721 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:58.389696    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxx4k\" (UniqueName: \"kubernetes.io/projected/2d6ab821-525f-41c1-8a94-c83c1b9147aa-kube-api-access-wxx4k\") pod \"test-curl-7612-f3df31\" (UID: \"2d6ab821-525f-41c1-8a94-c83c1b9147aa\") " pod="opendatahub/test-curl-7612-f3df31"
Sep 11 12:27:58.427580 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:58.427551    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-f3df31"
Sep 11 12:27:58.701075 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:58.701043    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-f3df31"]
Sep 11 12:27:58.703336 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:27:58.703298    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d6ab821_525f_41c1_8a94_c83c1b9147aa.slice/crio-2f318b0adc74aaeb77c70fffc4a59b156bbd0738e77ab37aa24a9963c2a58dbd WatchSource:0}: Error finding container 2f318b0adc74aaeb77c70fffc4a59b156bbd0738e77ab37aa24a9963c2a58dbd: Status 404 returned error can't find the container with id 2f318b0adc74aaeb77c70fffc4a59b156bbd0738e77ab37aa24a9963c2a58dbd
Sep 11 12:27:59.274443 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:59.274410    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-f3df31" event={"ID":"2d6ab821-525f-41c1-8a94-c83c1b9147aa","Type":"ContainerStarted","Data":"2f318b0adc74aaeb77c70fffc4a59b156bbd0738e77ab37aa24a9963c2a58dbd"}
Sep 11 12:27:59.929869 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:27:59.929832    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:28:01.286505 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:01.286467    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-f3df31" event={"ID":"2d6ab821-525f-41c1-8a94-c83c1b9147aa","Type":"ContainerStarted","Data":"fcca6d74bd1d66d3f52d8bc3eca38a893c06e2baf6809ef4f64f66d1af6dba55"}
Sep 11 12:28:01.307109 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:01.307062    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7612-f3df31" podStartSLOduration=1.084538189 podStartE2EDuration="3.307048389s" totalImagesPullingTime="2.2225102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:27:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:01.306498416 +0000 UTC m=+812.869278180" watchObservedRunningTime="2026-09-11 12:28:01.307048389 +0000 UTC m=+812.869828152"
Sep 11 12:28:01.889310 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:01.889273    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7612-f3df31"]
Sep 11 12:28:01.893803 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:01.893777    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7612-f3df31"]
Sep 11 12:28:02.012463 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:02.012433    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7612-cdbfaf"]
Sep 11 12:28:02.012758 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:02.012747    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d6ab821-525f-41c1-8a94-c83c1b9147aa" containerName="test-curl-7612-f3df31"
Sep 11 12:28:02.012805 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:02.012760    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d6ab821-525f-41c1-8a94-c83c1b9147aa" containerName="test-curl-7612-f3df31"
Sep 11 12:28:02.012839 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:02.012812    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d6ab821-525f-41c1-8a94-c83c1b9147aa" containerName="test-curl-7612-f3df31"
Sep 11 12:28:02.015851 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:02.015834    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-cdbfaf"
Sep 11 12:28:02.023709 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:02.023683    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-cdbfaf"]
Sep 11 12:28:02.097241 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:02.097184    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjtnk\" (UniqueName: \"kubernetes.io/projected/de63ba4d-6ea2-4878-89ad-9b1451334d84-kube-api-access-hjtnk\") pod \"test-curl-7612-cdbfaf\" (UID: \"de63ba4d-6ea2-4878-89ad-9b1451334d84\") " pod="opendatahub/test-curl-7612-cdbfaf"
Sep 11 12:28:02.197934 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:02.197889    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjtnk\" (UniqueName: \"kubernetes.io/projected/de63ba4d-6ea2-4878-89ad-9b1451334d84-kube-api-access-hjtnk\") pod \"test-curl-7612-cdbfaf\" (UID: \"de63ba4d-6ea2-4878-89ad-9b1451334d84\") " pod="opendatahub/test-curl-7612-cdbfaf"
Sep 11 12:28:02.210370 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:02.210342    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjtnk\" (UniqueName: \"kubernetes.io/projected/de63ba4d-6ea2-4878-89ad-9b1451334d84-kube-api-access-hjtnk\") pod \"test-curl-7612-cdbfaf\" (UID: \"de63ba4d-6ea2-4878-89ad-9b1451334d84\") " pod="opendatahub/test-curl-7612-cdbfaf"
Sep 11 12:28:02.290729 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:28:02.290657    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7612-f3df31" container="test-curl-7612-f3df31"
Sep 11 12:28:02.325975 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:02.325953    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-cdbfaf"
Sep 11 12:28:02.596135 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:02.596108    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-cdbfaf"]
Sep 11 12:28:02.598605 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:28:02.598565    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde63ba4d_6ea2_4878_89ad_9b1451334d84.slice/crio-633219076f89a2d9ae828001d0bdcc651ed40cb3e735f52cabfd24a9092320d5 WatchSource:0}: Error finding container 633219076f89a2d9ae828001d0bdcc651ed40cb3e735f52cabfd24a9092320d5: Status 404 returned error can't find the container with id 633219076f89a2d9ae828001d0bdcc651ed40cb3e735f52cabfd24a9092320d5
Sep 11 12:28:03.255687 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:03.255655    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb"]
Sep 11 12:28:03.258950 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:03.258933    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb"
Sep 11 12:28:03.261570 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:03.261543    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-38a724\""
Sep 11 12:28:03.261666 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:03.261628    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-38a724-dockercfg-9fdfs\""
Sep 11 12:28:03.269959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:03.269936    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb"]
Sep 11 12:28:03.296598 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:03.296574    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-cdbfaf" event={"ID":"de63ba4d-6ea2-4878-89ad-9b1451334d84","Type":"ContainerStarted","Data":"2b0f80daaaebf73c1ec0c095baf87b7605c63e2385989204029ceed7518c0543"}
Sep 11 12:28:03.296896 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:03.296607    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-cdbfaf" event={"ID":"de63ba4d-6ea2-4878-89ad-9b1451334d84","Type":"ContainerStarted","Data":"633219076f89a2d9ae828001d0bdcc651ed40cb3e735f52cabfd24a9092320d5"}
Sep 11 12:28:03.296896 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:03.296729    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7612-f3df31" podUID="2d6ab821-525f-41c1-8a94-c83c1b9147aa" containerName="test-curl-7612-f3df31" containerID="cri-o://fcca6d74bd1d66d3f52d8bc3eca38a893c06e2baf6809ef4f64f66d1af6dba55" gracePeriod=2
Sep 11 12:28:03.306749 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:03.306728    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2242f0-9987-4537-82ef-27d33f94c69e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb\" (UID: \"0a2242f0-9987-4537-82ef-27d33f94c69e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb"
Sep 11 12:28:03.306841 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:03.306762    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfz6h\" (UniqueName: \"kubernetes.io/projected/0a2242f0-9987-4537-82ef-27d33f94c69e-kube-api-access-bfz6h\") pod \"payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb\" (UID: \"0a2242f0-9987-4537-82ef-27d33f94c69e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb"
Sep 11 12:28:03.407958 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:03.407934    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2242f0-9987-4537-82ef-27d33f94c69e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb\" (UID: \"0a2242f0-9987-4537-82ef-27d33f94c69e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb"
Sep 11 12:28:03.408067 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:03.407963    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfz6h\" (UniqueName: \"kubernetes.io/projected/0a2242f0-9987-4537-82ef-27d33f94c69e-kube-api-access-bfz6h\") pod \"payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb\" (UID: \"0a2242f0-9987-4537-82ef-27d33f94c69e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb"
Sep 11 12:28:03.408702 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:03.408677    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2242f0-9987-4537-82ef-27d33f94c69e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb\" (UID: \"0a2242f0-9987-4537-82ef-27d33f94c69e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb"
Sep 11 12:28:03.418922 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:03.418895    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfz6h\" (UniqueName: \"kubernetes.io/projected/0a2242f0-9987-4537-82ef-27d33f94c69e-kube-api-access-bfz6h\") pod \"payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb\" (UID: \"0a2242f0-9987-4537-82ef-27d33f94c69e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb"
Sep 11 12:28:03.569873 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:03.569805    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb"
Sep 11 12:28:03.838815 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:03.838100    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7612-cdbfaf" podStartSLOduration=1.8380805169999999 podStartE2EDuration="1.838080517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:03.315657164 +0000 UTC m=+814.878436927" watchObservedRunningTime="2026-09-11 12:28:03.838080517 +0000 UTC m=+815.400860280"
Sep 11 12:28:03.840040 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:03.840020    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb"]
Sep 11 12:28:03.842458 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:28:03.842434    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2242f0_9987_4537_82ef_27d33f94c69e.slice/crio-96b89ee7f0b4860774145d03bc093ad8ef1c4732c3e1ce8a4371239e9136382b WatchSource:0}: Error finding container 96b89ee7f0b4860774145d03bc093ad8ef1c4732c3e1ce8a4371239e9136382b: Status 404 returned error can't find the container with id 96b89ee7f0b4860774145d03bc093ad8ef1c4732c3e1ce8a4371239e9136382b
Sep 11 12:28:03.923850 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:03.923823    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7612-cdbfaf"]
Sep 11 12:28:03.930016 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:03.929987    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7612-cdbfaf"]
Sep 11 12:28:04.308128 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:04.308095    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb" event={"ID":"0a2242f0-9987-4537-82ef-27d33f94c69e","Type":"ContainerStarted","Data":"96b89ee7f0b4860774145d03bc093ad8ef1c4732c3e1ce8a4371239e9136382b"}
Sep 11 12:28:04.308500 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:28:04.308294    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7612-cdbfaf" container="test-curl-7612-cdbfaf"
Sep 11 12:28:05.313652 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:05.313622    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7612-cdbfaf" podUID="de63ba4d-6ea2-4878-89ad-9b1451334d84" containerName="test-curl-7612-cdbfaf" containerID="cri-o://2b0f80daaaebf73c1ec0c095baf87b7605c63e2385989204029ceed7518c0543" gracePeriod=2
Sep 11 12:28:05.572423 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:05.572397    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-f3df31"
Sep 11 12:28:05.626611 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:05.626578    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxx4k\" (UniqueName: \"kubernetes.io/projected/2d6ab821-525f-41c1-8a94-c83c1b9147aa-kube-api-access-wxx4k\") pod \"2d6ab821-525f-41c1-8a94-c83c1b9147aa\" (UID: \"2d6ab821-525f-41c1-8a94-c83c1b9147aa\") "
Sep 11 12:28:05.628679 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:05.628647    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6ab821-525f-41c1-8a94-c83c1b9147aa-kube-api-access-wxx4k" (OuterVolumeSpecName: "kube-api-access-wxx4k") pod "2d6ab821-525f-41c1-8a94-c83c1b9147aa" (UID: "2d6ab821-525f-41c1-8a94-c83c1b9147aa"). InnerVolumeSpecName "kube-api-access-wxx4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:28:05.727863 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:05.727828    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxx4k\" (UniqueName: \"kubernetes.io/projected/2d6ab821-525f-41c1-8a94-c83c1b9147aa-kube-api-access-wxx4k\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:28:05.957836 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:05.957764    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk"]
Sep 11 12:28:05.958101 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:05.958088    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de63ba4d-6ea2-4878-89ad-9b1451334d84" containerName="test-curl-7612-cdbfaf"
Sep 11 12:28:05.958148 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:05.958102    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="de63ba4d-6ea2-4878-89ad-9b1451334d84" containerName="test-curl-7612-cdbfaf"
Sep 11 12:28:05.958206 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:05.958163    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="de63ba4d-6ea2-4878-89ad-9b1451334d84" containerName="test-curl-7612-cdbfaf"
Sep 11 12:28:05.961059 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:05.961042    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk"
Sep 11 12:28:05.963422 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:05.963401    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-cb20c8-dockercfg-5dnsg\""
Sep 11 12:28:05.963666 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:05.963652    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-cb20c8\""
Sep 11 12:28:05.972574 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:05.972552    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk"]
Sep 11 12:28:05.991157 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:05.991134    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5"]
Sep 11 12:28:05.994369 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:05.994355    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5"
Sep 11 12:28:06.006611 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.006589    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5"]
Sep 11 12:28:06.029982 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.029953    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc6bq\" (UniqueName: \"kubernetes.io/projected/6d2a4d3c-95d8-4dec-b576-9ef468ffcf42-kube-api-access-xc6bq\") pod \"payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk\" (UID: \"6d2a4d3c-95d8-4dec-b576-9ef468ffcf42\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk"
Sep 11 12:28:06.030078 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.030008    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2a4d3c-95d8-4dec-b576-9ef468ffcf42-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk\" (UID: \"6d2a4d3c-95d8-4dec-b576-9ef468ffcf42\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk"
Sep 11 12:28:06.030135 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.030104    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f49fb2d-fa85-4cba-a555-6e455299cb06-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5\" (UID: \"3f49fb2d-fa85-4cba-a555-6e455299cb06\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5"
Sep 11 12:28:06.030184 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.030170    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79hkj\" (UniqueName: \"kubernetes.io/projected/3f49fb2d-fa85-4cba-a555-6e455299cb06-kube-api-access-79hkj\") pod \"payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5\" (UID: \"3f49fb2d-fa85-4cba-a555-6e455299cb06\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5"
Sep 11 12:28:06.131511 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.131484    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc6bq\" (UniqueName: \"kubernetes.io/projected/6d2a4d3c-95d8-4dec-b576-9ef468ffcf42-kube-api-access-xc6bq\") pod \"payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk\" (UID: \"6d2a4d3c-95d8-4dec-b576-9ef468ffcf42\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk"
Sep 11 12:28:06.131654 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.131531    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2a4d3c-95d8-4dec-b576-9ef468ffcf42-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk\" (UID: \"6d2a4d3c-95d8-4dec-b576-9ef468ffcf42\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk"
Sep 11 12:28:06.131654 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.131566    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f49fb2d-fa85-4cba-a555-6e455299cb06-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5\" (UID: \"3f49fb2d-fa85-4cba-a555-6e455299cb06\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5"
Sep 11 12:28:06.131654 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.131606    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79hkj\" (UniqueName: \"kubernetes.io/projected/3f49fb2d-fa85-4cba-a555-6e455299cb06-kube-api-access-79hkj\") pod \"payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5\" (UID: \"3f49fb2d-fa85-4cba-a555-6e455299cb06\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5"
Sep 11 12:28:06.132300 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.132277    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2a4d3c-95d8-4dec-b576-9ef468ffcf42-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk\" (UID: \"6d2a4d3c-95d8-4dec-b576-9ef468ffcf42\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk"
Sep 11 12:28:06.132470 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.132449    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f49fb2d-fa85-4cba-a555-6e455299cb06-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5\" (UID: \"3f49fb2d-fa85-4cba-a555-6e455299cb06\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5"
Sep 11 12:28:06.139055 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.139034    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79hkj\" (UniqueName: \"kubernetes.io/projected/3f49fb2d-fa85-4cba-a555-6e455299cb06-kube-api-access-79hkj\") pod \"payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5\" (UID: \"3f49fb2d-fa85-4cba-a555-6e455299cb06\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5"
Sep 11 12:28:06.139130 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.139056    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc6bq\" (UniqueName: \"kubernetes.io/projected/6d2a4d3c-95d8-4dec-b576-9ef468ffcf42-kube-api-access-xc6bq\") pod \"payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk\" (UID: \"6d2a4d3c-95d8-4dec-b576-9ef468ffcf42\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk"
Sep 11 12:28:06.271726 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.271698    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk"
Sep 11 12:28:06.303562 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.303536    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5"
Sep 11 12:28:06.324053 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.323672    2561 generic.go:356] "Generic (PLEG): container finished" podID="2d6ab821-525f-41c1-8a94-c83c1b9147aa" containerID="fcca6d74bd1d66d3f52d8bc3eca38a893c06e2baf6809ef4f64f66d1af6dba55" exitCode=137
Sep 11 12:28:06.324053 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.323877    2561 scope.go:117] "RemoveContainer" containerID="fcca6d74bd1d66d3f52d8bc3eca38a893c06e2baf6809ef4f64f66d1af6dba55"
Sep 11 12:28:06.324053 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.323906    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-f3df31"
Sep 11 12:28:06.332901 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.331868    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb" event={"ID":"0a2242f0-9987-4537-82ef-27d33f94c69e","Type":"ContainerStarted","Data":"08356ed73342bfc424ae8ca536fdb34295c9916e76e272add07ccf999587b00a"}
Sep 11 12:28:06.332901 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.332806    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb"
Sep 11 12:28:06.340846 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.340826    2561 scope.go:117] "RemoveContainer" containerID="fcca6d74bd1d66d3f52d8bc3eca38a893c06e2baf6809ef4f64f66d1af6dba55"
Sep 11 12:28:06.341254 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:28:06.341172    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcca6d74bd1d66d3f52d8bc3eca38a893c06e2baf6809ef4f64f66d1af6dba55\": container with ID starting with fcca6d74bd1d66d3f52d8bc3eca38a893c06e2baf6809ef4f64f66d1af6dba55 not found: ID does not exist" containerID="fcca6d74bd1d66d3f52d8bc3eca38a893c06e2baf6809ef4f64f66d1af6dba55"
Sep 11 12:28:06.341254 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.341235    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcca6d74bd1d66d3f52d8bc3eca38a893c06e2baf6809ef4f64f66d1af6dba55"} err="failed to get container status \"fcca6d74bd1d66d3f52d8bc3eca38a893c06e2baf6809ef4f64f66d1af6dba55\": rpc error: code = NotFound desc = could not find container \"fcca6d74bd1d66d3f52d8bc3eca38a893c06e2baf6809ef4f64f66d1af6dba55\": container with ID starting with fcca6d74bd1d66d3f52d8bc3eca38a893c06e2baf6809ef4f64f66d1af6dba55 not found: ID does not exist"
Sep 11 12:28:06.354234 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.354127    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb" podStartSLOduration=1.707544139 podStartE2EDuration="3.35410817s" totalImagesPullingTime="1.646564031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:06.353370801 +0000 UTC m=+817.916150566" watchObservedRunningTime="2026-09-11 12:28:06.35410817 +0000 UTC m=+817.916887934"
Sep 11 12:28:06.356374 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:28:06.356343    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7612-f3df31\" is forbidden: User \"system:node:ip-10-0-131-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-172.ec2.internal' and this object" podUID="2d6ab821-525f-41c1-8a94-c83c1b9147aa" pod="opendatahub/test-curl-7612-f3df31"
Sep 11 12:28:06.544845 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.544816    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk"]
Sep 11 12:28:06.546600 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:28:06.546564    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d2a4d3c_95d8_4dec_b576_9ef468ffcf42.slice/crio-24974b293c3939fa40fd94f0be34396dee5b337c25470035427edfeaa69835e0 WatchSource:0}: Error finding container 24974b293c3939fa40fd94f0be34396dee5b337c25470035427edfeaa69835e0: Status 404 returned error can't find the container with id 24974b293c3939fa40fd94f0be34396dee5b337c25470035427edfeaa69835e0
Sep 11 12:28:06.591086 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.591057    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5"]
Sep 11 12:28:06.593957 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:28:06.593931    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f49fb2d_fa85_4cba_a555_6e455299cb06.slice/crio-0c6b4d1f66795eaad9fae0d9f7db044263ebe164e44f9fd64cbd5fb2ca2c63e1 WatchSource:0}: Error finding container 0c6b4d1f66795eaad9fae0d9f7db044263ebe164e44f9fd64cbd5fb2ca2c63e1: Status 404 returned error can't find the container with id 0c6b4d1f66795eaad9fae0d9f7db044263ebe164e44f9fd64cbd5fb2ca2c63e1
Sep 11 12:28:06.933492 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:06.933400    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d6ab821-525f-41c1-8a94-c83c1b9147aa" path="/var/lib/kubelet/pods/2d6ab821-525f-41c1-8a94-c83c1b9147aa/volumes"
Sep 11 12:28:07.341706 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:07.341674    2561 generic.go:356] "Generic (PLEG): container finished" podID="de63ba4d-6ea2-4878-89ad-9b1451334d84" containerID="2b0f80daaaebf73c1ec0c095baf87b7605c63e2385989204029ceed7518c0543" exitCode=137
Sep 11 12:28:07.344813 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:07.344782    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5" event={"ID":"3f49fb2d-fa85-4cba-a555-6e455299cb06","Type":"ContainerStarted","Data":"4cefa318e39077adf21a703fe5efb4e5cfc3f51aa0e91059ba98e2c8427afee2"}
Sep 11 12:28:07.344943 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:07.344845    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5" event={"ID":"3f49fb2d-fa85-4cba-a555-6e455299cb06","Type":"ContainerStarted","Data":"0c6b4d1f66795eaad9fae0d9f7db044263ebe164e44f9fd64cbd5fb2ca2c63e1"}
Sep 11 12:28:07.345005 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:07.344942    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5"
Sep 11 12:28:07.346577 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:07.346552    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk" event={"ID":"6d2a4d3c-95d8-4dec-b576-9ef468ffcf42","Type":"ContainerStarted","Data":"b0b90dbca3f9fb27a6b87a5d3655df9d85a2e8541fa74e0f2933289f46eeacf2"}
Sep 11 12:28:07.346694 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:07.346598    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk" event={"ID":"6d2a4d3c-95d8-4dec-b576-9ef468ffcf42","Type":"ContainerStarted","Data":"24974b293c3939fa40fd94f0be34396dee5b337c25470035427edfeaa69835e0"}
Sep 11 12:28:07.346825 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:07.346806    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk"
Sep 11 12:28:07.362040 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:07.361998    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5" podStartSLOduration=2.36198248 podStartE2EDuration="2.36198248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:07.360619611 +0000 UTC m=+818.923399374" watchObservedRunningTime="2026-09-11 12:28:07.36198248 +0000 UTC m=+818.924762243"
Sep 11 12:28:07.378671 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:07.378627    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk" podStartSLOduration=2.378613972 podStartE2EDuration="2.378613972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:07.377655941 +0000 UTC m=+818.940435704" watchObservedRunningTime="2026-09-11 12:28:07.378613972 +0000 UTC m=+818.941393735"
Sep 11 12:28:07.464553 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:07.464529    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-cdbfaf"
Sep 11 12:28:07.545800 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:07.545767    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjtnk\" (UniqueName: \"kubernetes.io/projected/de63ba4d-6ea2-4878-89ad-9b1451334d84-kube-api-access-hjtnk\") pod \"de63ba4d-6ea2-4878-89ad-9b1451334d84\" (UID: \"de63ba4d-6ea2-4878-89ad-9b1451334d84\") "
Sep 11 12:28:07.547891 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:07.547864    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de63ba4d-6ea2-4878-89ad-9b1451334d84-kube-api-access-hjtnk" (OuterVolumeSpecName: "kube-api-access-hjtnk") pod "de63ba4d-6ea2-4878-89ad-9b1451334d84" (UID: "de63ba4d-6ea2-4878-89ad-9b1451334d84"). InnerVolumeSpecName "kube-api-access-hjtnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:28:07.646904 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:07.646824    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjtnk\" (UniqueName: \"kubernetes.io/projected/de63ba4d-6ea2-4878-89ad-9b1451334d84-kube-api-access-hjtnk\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:28:08.352386 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:08.352352    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-cdbfaf"
Sep 11 12:28:08.352386 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:08.352354    2561 scope.go:117] "RemoveContainer" containerID="2b0f80daaaebf73c1ec0c095baf87b7605c63e2385989204029ceed7518c0543"
Sep 11 12:28:08.368057 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:28:08.368026    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7612-cdbfaf\" is forbidden: User \"system:node:ip-10-0-131-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-172.ec2.internal' and this object" podUID="de63ba4d-6ea2-4878-89ad-9b1451334d84" pod="opendatahub/test-curl-7612-cdbfaf"
Sep 11 12:28:08.930502 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:28:08.930462    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7612-cdbfaf\" is forbidden: User \"system:node:ip-10-0-131-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-172.ec2.internal' and this object" podUID="de63ba4d-6ea2-4878-89ad-9b1451334d84" pod="opendatahub/test-curl-7612-cdbfaf"
Sep 11 12:28:08.932258 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:08.932234    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de63ba4d-6ea2-4878-89ad-9b1451334d84" path="/var/lib/kubelet/pods/de63ba4d-6ea2-4878-89ad-9b1451334d84/volumes"
Sep 11 12:28:09.264946 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:09.264883    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9"]
Sep 11 12:28:09.269842 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:09.269819    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9"
Sep 11 12:28:09.272278 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:09.272254    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-9226c5-serving-cert\""
Sep 11 12:28:09.272404 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:09.272355    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 12:28:09.272471 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:09.272415    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 12:28:09.273284 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:09.273252    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 12:28:09.273372 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:09.273302    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l5tx4\""
Sep 11 12:28:09.279880 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:09.279859    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9"]
Sep 11 12:28:09.360681 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:09.360647    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53eb3c50-53be-4a66-8e89-b548806b50b4-maas-api-tls\") pod \"maas-api-e2e-autores-9226c5-788c69c596-qdkt9\" (UID: \"53eb3c50-53be-4a66-8e89-b548806b50b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9"
Sep 11 12:28:09.361033 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:09.360697    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53eb3c50-53be-4a66-8e89-b548806b50b4-openshift-service-ca\") pod \"maas-api-e2e-autores-9226c5-788c69c596-qdkt9\" (UID: \"53eb3c50-53be-4a66-8e89-b548806b50b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9"
Sep 11 12:28:09.361033 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:09.360787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkwzz\" (UniqueName: \"kubernetes.io/projected/53eb3c50-53be-4a66-8e89-b548806b50b4-kube-api-access-hkwzz\") pod \"maas-api-e2e-autores-9226c5-788c69c596-qdkt9\" (UID: \"53eb3c50-53be-4a66-8e89-b548806b50b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9"
Sep 11 12:28:09.361033 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:09.360833    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53eb3c50-53be-4a66-8e89-b548806b50b4-metrics-certs\") pod \"maas-api-e2e-autores-9226c5-788c69c596-qdkt9\" (UID: \"53eb3c50-53be-4a66-8e89-b548806b50b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9"
Sep 11 12:28:09.461314 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:09.461281    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53eb3c50-53be-4a66-8e89-b548806b50b4-metrics-certs\") pod \"maas-api-e2e-autores-9226c5-788c69c596-qdkt9\" (UID: \"53eb3c50-53be-4a66-8e89-b548806b50b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9"
Sep 11 12:28:09.461460 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:09.461337    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53eb3c50-53be-4a66-8e89-b548806b50b4-maas-api-tls\") pod \"maas-api-e2e-autores-9226c5-788c69c596-qdkt9\" (UID: \"53eb3c50-53be-4a66-8e89-b548806b50b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9"
Sep 11 12:28:09.461460 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:09.461363    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53eb3c50-53be-4a66-8e89-b548806b50b4-openshift-service-ca\") pod \"maas-api-e2e-autores-9226c5-788c69c596-qdkt9\" (UID: \"53eb3c50-53be-4a66-8e89-b548806b50b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9"
Sep 11 12:28:09.461460 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:09.461393    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkwzz\" (UniqueName: \"kubernetes.io/projected/53eb3c50-53be-4a66-8e89-b548806b50b4-kube-api-access-hkwzz\") pod \"maas-api-e2e-autores-9226c5-788c69c596-qdkt9\" (UID: \"53eb3c50-53be-4a66-8e89-b548806b50b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9"
Sep 11 12:28:09.462045 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:09.462021    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53eb3c50-53be-4a66-8e89-b548806b50b4-openshift-service-ca\") pod \"maas-api-e2e-autores-9226c5-788c69c596-qdkt9\" (UID: \"53eb3c50-53be-4a66-8e89-b548806b50b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9"
Sep 11 12:28:09.463873 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:09.463847    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53eb3c50-53be-4a66-8e89-b548806b50b4-maas-api-tls\") pod \"maas-api-e2e-autores-9226c5-788c69c596-qdkt9\" (UID: \"53eb3c50-53be-4a66-8e89-b548806b50b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9"
Sep 11 12:28:09.463956 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:09.463849    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53eb3c50-53be-4a66-8e89-b548806b50b4-metrics-certs\") pod \"maas-api-e2e-autores-9226c5-788c69c596-qdkt9\" (UID: \"53eb3c50-53be-4a66-8e89-b548806b50b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9"
Sep 11 12:28:09.469724 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:09.469699    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkwzz\" (UniqueName: \"kubernetes.io/projected/53eb3c50-53be-4a66-8e89-b548806b50b4-kube-api-access-hkwzz\") pod \"maas-api-e2e-autores-9226c5-788c69c596-qdkt9\" (UID: \"53eb3c50-53be-4a66-8e89-b548806b50b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9"
Sep 11 12:28:09.581765 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:09.581704    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9"
Sep 11 12:28:09.853987 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:09.853961    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9"]
Sep 11 12:28:09.855904 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:28:09.855877    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53eb3c50_53be_4a66_8e89_b548806b50b4.slice/crio-a4ec61466cd22461ac95cf5a11e562182de7154747b80251609a95663f3d65b2 WatchSource:0}: Error finding container a4ec61466cd22461ac95cf5a11e562182de7154747b80251609a95663f3d65b2: Status 404 returned error can't find the container with id a4ec61466cd22461ac95cf5a11e562182de7154747b80251609a95663f3d65b2
Sep 11 12:28:10.362633 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:10.362605    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9" event={"ID":"53eb3c50-53be-4a66-8e89-b548806b50b4","Type":"ContainerStarted","Data":"a4ec61466cd22461ac95cf5a11e562182de7154747b80251609a95663f3d65b2"}
Sep 11 12:28:11.086785 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:11.086748    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl"]
Sep 11 12:28:11.090125 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:11.090110    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl"
Sep 11 12:28:11.092618 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:11.092599    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7e8d45b5\""
Sep 11 12:28:11.092756 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:11.092739    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7e8d45b5-dockercfg-xwf94\""
Sep 11 12:28:11.105327 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:11.105301    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl"]
Sep 11 12:28:11.121037 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:11.121012    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh"]
Sep 11 12:28:11.124024 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:11.124008    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh"
Sep 11 12:28:11.134645 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:11.134619    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh"]
Sep 11 12:28:11.175797 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:11.175768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f5rx\" (UniqueName: \"kubernetes.io/projected/8b526270-8769-48c5-a0cd-a01aeadefd3d-kube-api-access-6f5rx\") pod \"payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl\" (UID: \"8b526270-8769-48c5-a0cd-a01aeadefd3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl"
Sep 11 12:28:11.175926 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:11.175814    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b526270-8769-48c5-a0cd-a01aeadefd3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl\" (UID: \"8b526270-8769-48c5-a0cd-a01aeadefd3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl"
Sep 11 12:28:11.175926 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:11.175881    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnggm\" (UniqueName: \"kubernetes.io/projected/2b9c8f9b-7b7a-4cc9-acea-956654d6d732-kube-api-access-jnggm\") pod \"payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh\" (UID: \"2b9c8f9b-7b7a-4cc9-acea-956654d6d732\") " pod="openshift-ingress/payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh"
Sep 11 12:28:11.176032 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:11.175961    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9c8f9b-7b7a-4cc9-acea-956654d6d732-plugins-config-volume\") pod \"payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh\" (UID: \"2b9c8f9b-7b7a-4cc9-acea-956654d6d732\") " pod="openshift-ingress/payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh"
Sep 11 12:28:11.276981 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:11.276952    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f5rx\" (UniqueName: \"kubernetes.io/projected/8b526270-8769-48c5-a0cd-a01aeadefd3d-kube-api-access-6f5rx\") pod \"payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl\" (UID: \"8b526270-8769-48c5-a0cd-a01aeadefd3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl"
Sep 11 12:28:11.277100 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:11.276987    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b526270-8769-48c5-a0cd-a01aeadefd3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl\" (UID: \"8b526270-8769-48c5-a0cd-a01aeadefd3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl"
Sep 11 12:28:11.277100 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:11.277013    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnggm\" (UniqueName: \"kubernetes.io/projected/2b9c8f9b-7b7a-4cc9-acea-956654d6d732-kube-api-access-jnggm\") pod \"payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh\" (UID: \"2b9c8f9b-7b7a-4cc9-acea-956654d6d732\") " pod="openshift-ingress/payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh"
Sep 11 12:28:11.277100 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:11.277054    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9c8f9b-7b7a-4cc9-acea-956654d6d732-plugins-config-volume\") pod \"payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh\" (UID: \"2b9c8f9b-7b7a-4cc9-acea-956654d6d732\") " pod="openshift-ingress/payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh"
Sep 11 12:28:11.277722 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:11.277704    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9c8f9b-7b7a-4cc9-acea-956654d6d732-plugins-config-volume\") pod \"payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh\" (UID: \"2b9c8f9b-7b7a-4cc9-acea-956654d6d732\") " pod="openshift-ingress/payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh"
Sep 11 12:28:11.277776 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:11.277720    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b526270-8769-48c5-a0cd-a01aeadefd3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl\" (UID: \"8b526270-8769-48c5-a0cd-a01aeadefd3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl"
Sep 11 12:28:11.285563 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:11.285542    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnggm\" (UniqueName: \"kubernetes.io/projected/2b9c8f9b-7b7a-4cc9-acea-956654d6d732-kube-api-access-jnggm\") pod \"payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh\" (UID: \"2b9c8f9b-7b7a-4cc9-acea-956654d6d732\") " pod="openshift-ingress/payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh"
Sep 11 12:28:11.285735 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:11.285718    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f5rx\" (UniqueName: \"kubernetes.io/projected/8b526270-8769-48c5-a0cd-a01aeadefd3d-kube-api-access-6f5rx\") pod \"payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl\" (UID: \"8b526270-8769-48c5-a0cd-a01aeadefd3d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl"
Sep 11 12:28:11.400280 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:11.400207    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl"
Sep 11 12:28:11.433459 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:11.433434    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh"
Sep 11 12:28:11.672539 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:11.672513    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl"]
Sep 11 12:28:11.674476 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:28:11.674442    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b526270_8769_48c5_a0cd_a01aeadefd3d.slice/crio-8526190092eb6bfc10cb03bd774775674473e731d56ec7dd3435a7fd7d8098ad WatchSource:0}: Error finding container 8526190092eb6bfc10cb03bd774775674473e731d56ec7dd3435a7fd7d8098ad: Status 404 returned error can't find the container with id 8526190092eb6bfc10cb03bd774775674473e731d56ec7dd3435a7fd7d8098ad
Sep 11 12:28:11.726008 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:11.725980    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh"]
Sep 11 12:28:11.727655 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:28:11.727630    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9c8f9b_7b7a_4cc9_acea_956654d6d732.slice/crio-4e039badff5cb30dd6d3099fa01a21b59f63a2aee1faff6bd2aac5c1202daefb WatchSource:0}: Error finding container 4e039badff5cb30dd6d3099fa01a21b59f63a2aee1faff6bd2aac5c1202daefb: Status 404 returned error can't find the container with id 4e039badff5cb30dd6d3099fa01a21b59f63a2aee1faff6bd2aac5c1202daefb
Sep 11 12:28:12.374082 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:12.374042    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl" event={"ID":"8b526270-8769-48c5-a0cd-a01aeadefd3d","Type":"ContainerStarted","Data":"2fb964aa302ad3c1adc53af21b654f8a5cb7942e4e7c549c7db64dd9bc87e81c"}
Sep 11 12:28:12.374262 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:12.374154    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl" event={"ID":"8b526270-8769-48c5-a0cd-a01aeadefd3d","Type":"ContainerStarted","Data":"8526190092eb6bfc10cb03bd774775674473e731d56ec7dd3435a7fd7d8098ad"}
Sep 11 12:28:12.374262 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:12.374205    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl"
Sep 11 12:28:12.376046 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:12.376012    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh" event={"ID":"2b9c8f9b-7b7a-4cc9-acea-956654d6d732","Type":"ContainerStarted","Data":"255bf4a8df31146bdfa14419b2b6f6d59ea927a88a91341b3246261e083cc058"}
Sep 11 12:28:12.376174 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:12.376075    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh" event={"ID":"2b9c8f9b-7b7a-4cc9-acea-956654d6d732","Type":"ContainerStarted","Data":"4e039badff5cb30dd6d3099fa01a21b59f63a2aee1faff6bd2aac5c1202daefb"}
Sep 11 12:28:12.376174 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:12.376138    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh"
Sep 11 12:28:12.397034 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:12.396988    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl" podStartSLOduration=1.39696841 podStartE2EDuration="1.39696841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:12.395450232 +0000 UTC m=+823.958229995" watchObservedRunningTime="2026-09-11 12:28:12.39696841 +0000 UTC m=+823.959748175"
Sep 11 12:28:12.412471 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:12.412421    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh" podStartSLOduration=1.412402256 podStartE2EDuration="1.412402256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:12.41115351 +0000 UTC m=+823.973933275" watchObservedRunningTime="2026-09-11 12:28:12.412402256 +0000 UTC m=+823.975182020"
Sep 11 12:28:14.387136 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:14.387105    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9" event={"ID":"53eb3c50-53be-4a66-8e89-b548806b50b4","Type":"ContainerStarted","Data":"82afa211f8c6f36fb79bd7eb7e29745063ce392ea57fcfb69611e9efa6afc81f"}
Sep 11 12:28:14.387583 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:14.387176    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9"
Sep 11 12:28:14.404930 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:14.404885    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9" podStartSLOduration=1.8163005540000001 podStartE2EDuration="5.404870083s" totalImagesPullingTime="3.588569529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:14.403384296 +0000 UTC m=+825.966164064" watchObservedRunningTime="2026-09-11 12:28:14.404870083 +0000 UTC m=+825.967650111"
Sep 11 12:28:15.249723 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:15.249681    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:28:18.354496 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:18.354466    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb"
Sep 11 12:28:18.354871 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:18.354567    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk"
Sep 11 12:28:18.354912 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:18.354870    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5"
Sep 11 12:28:20.397964 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:20.397934    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9"
Sep 11 12:28:23.382943 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:23.382910    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh"
Sep 11 12:28:23.383333 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:23.383069    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl"
Sep 11 12:28:31.007965 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:31.007932    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"]
Sep 11 12:28:31.012744 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:31.012724    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"
Sep 11 12:28:31.015077 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:31.015055    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-edffad-serving-cert\""
Sep 11 12:28:31.024092 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:31.024069    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"]
Sep 11 12:28:31.157642 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:31.157605    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-metrics-certs\") pod \"maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b\" (UID: \"483dc020-9fa7-4bea-98b4-5c65c9adf3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"
Sep 11 12:28:31.157820 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:31.157678    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cttkt\" (UniqueName: \"kubernetes.io/projected/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-kube-api-access-cttkt\") pod \"maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b\" (UID: \"483dc020-9fa7-4bea-98b4-5c65c9adf3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"
Sep 11 12:28:31.157820 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:31.157717    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b\" (UID: \"483dc020-9fa7-4bea-98b4-5c65c9adf3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"
Sep 11 12:28:31.157820 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:31.157781    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-maas-api-tls\") pod \"maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b\" (UID: \"483dc020-9fa7-4bea-98b4-5c65c9adf3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"
Sep 11 12:28:31.258443 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:31.258356    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-metrics-certs\") pod \"maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b\" (UID: \"483dc020-9fa7-4bea-98b4-5c65c9adf3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"
Sep 11 12:28:31.258443 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:31.258421    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cttkt\" (UniqueName: \"kubernetes.io/projected/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-kube-api-access-cttkt\") pod \"maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b\" (UID: \"483dc020-9fa7-4bea-98b4-5c65c9adf3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"
Sep 11 12:28:31.258739 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:31.258455    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b\" (UID: \"483dc020-9fa7-4bea-98b4-5c65c9adf3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"
Sep 11 12:28:31.258739 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:31.258494    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-maas-api-tls\") pod \"maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b\" (UID: \"483dc020-9fa7-4bea-98b4-5c65c9adf3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"
Sep 11 12:28:31.258739 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:28:31.258627    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-serving-cert: secret "maas-api-e2e-inf-b-edffad-serving-cert" not found
Sep 11 12:28:31.258739 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:28:31.258700    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-maas-api-tls podName:483dc020-9fa7-4bea-98b4-5c65c9adf3e4 nodeName:}" failed. No retries permitted until 2026-09-11 12:28:31.758679679 +0000 UTC m=+843.321459420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-maas-api-tls") pod "maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b" (UID: "483dc020-9fa7-4bea-98b4-5c65c9adf3e4") : secret "maas-api-e2e-inf-b-edffad-serving-cert" not found
Sep 11 12:28:31.259077 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:31.259056    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b\" (UID: \"483dc020-9fa7-4bea-98b4-5c65c9adf3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"
Sep 11 12:28:31.260753 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:31.260736    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-metrics-certs\") pod \"maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b\" (UID: \"483dc020-9fa7-4bea-98b4-5c65c9adf3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"
Sep 11 12:28:31.266766 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:31.266740    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cttkt\" (UniqueName: \"kubernetes.io/projected/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-kube-api-access-cttkt\") pod \"maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b\" (UID: \"483dc020-9fa7-4bea-98b4-5c65c9adf3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"
Sep 11 12:28:31.763673 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:31.763640    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-maas-api-tls\") pod \"maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b\" (UID: \"483dc020-9fa7-4bea-98b4-5c65c9adf3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"
Sep 11 12:28:31.765985 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:31.765959    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-maas-api-tls\") pod \"maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b\" (UID: \"483dc020-9fa7-4bea-98b4-5c65c9adf3e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"
Sep 11 12:28:31.924885 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:31.924852    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"
Sep 11 12:28:32.197455 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:32.197428    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"]
Sep 11 12:28:32.199582 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:28:32.199550    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod483dc020_9fa7_4bea_98b4_5c65c9adf3e4.slice/crio-a715aa15b449cc2b6f0649788175b50728f0142222c2a9a2a6eb3fc025ae5778 WatchSource:0}: Error finding container a715aa15b449cc2b6f0649788175b50728f0142222c2a9a2a6eb3fc025ae5778: Status 404 returned error can't find the container with id a715aa15b449cc2b6f0649788175b50728f0142222c2a9a2a6eb3fc025ae5778
Sep 11 12:28:32.471155 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:32.471069    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b" event={"ID":"483dc020-9fa7-4bea-98b4-5c65c9adf3e4","Type":"ContainerStarted","Data":"a715aa15b449cc2b6f0649788175b50728f0142222c2a9a2a6eb3fc025ae5778"}
Sep 11 12:28:33.477504 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:33.477466    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b" event={"ID":"483dc020-9fa7-4bea-98b4-5c65c9adf3e4","Type":"ContainerStarted","Data":"1e012e995d6c5411a6b94b44f8d71c1c7463e7efa55d06b1159c5d03e6ff77cf"}
Sep 11 12:28:33.477865 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:33.477557    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"
Sep 11 12:28:33.497686 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:33.497631    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b" podStartSLOduration=2.978244991 podStartE2EDuration="3.497614028s" totalImagesPullingTime="519.369037ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:33.497557257 +0000 UTC m=+845.060337025" watchObservedRunningTime="2026-09-11 12:28:33.497614028 +0000 UTC m=+845.060393794"
Sep 11 12:28:37.533935 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:37.533901    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:28:39.487450 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:39.487416    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"
Sep 11 12:28:42.381715 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.381681    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"]
Sep 11 12:28:42.386741 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.386724    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:28:42.390442 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.390413    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1a8ee3\"/\"default-dockercfg-kmx7t\""
Sep 11 12:28:42.390572 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.390427    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1a8ee3\"/\"kube-root-ca.crt\""
Sep 11 12:28:42.390572 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.390429    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1a8ee3\"/\"openshift-service-ca.crt\""
Sep 11 12:28:42.390672 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.390427    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1a8ee3\"/\"simulator-w0-1a8ee3-kserve-self-signed-certs\""
Sep 11 12:28:42.394885 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.394864    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"]
Sep 11 12:28:42.565255 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.565220    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b502dcc5-5ffe-451f-9781-efcb69394137-dshm\") pod \"simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d\" (UID: \"b502dcc5-5ffe-451f-9781-efcb69394137\") " pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:28:42.565462 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.565316    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b502dcc5-5ffe-451f-9781-efcb69394137-tmp-dir\") pod \"simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d\" (UID: \"b502dcc5-5ffe-451f-9781-efcb69394137\") " pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:28:42.565462 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.565351    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b502dcc5-5ffe-451f-9781-efcb69394137-tls-certs\") pod \"simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d\" (UID: \"b502dcc5-5ffe-451f-9781-efcb69394137\") " pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:28:42.565462 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.565374    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b502dcc5-5ffe-451f-9781-efcb69394137-model-cache\") pod \"simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d\" (UID: \"b502dcc5-5ffe-451f-9781-efcb69394137\") " pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:28:42.565622 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.565481    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt68l\" (UniqueName: \"kubernetes.io/projected/b502dcc5-5ffe-451f-9781-efcb69394137-kube-api-access-gt68l\") pod \"simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d\" (UID: \"b502dcc5-5ffe-451f-9781-efcb69394137\") " pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:28:42.565622 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.565525    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b502dcc5-5ffe-451f-9781-efcb69394137-home\") pod \"simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d\" (UID: \"b502dcc5-5ffe-451f-9781-efcb69394137\") " pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:28:42.666671 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.666593    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b502dcc5-5ffe-451f-9781-efcb69394137-tmp-dir\") pod \"simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d\" (UID: \"b502dcc5-5ffe-451f-9781-efcb69394137\") " pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:28:42.666671 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.666625    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b502dcc5-5ffe-451f-9781-efcb69394137-tls-certs\") pod \"simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d\" (UID: \"b502dcc5-5ffe-451f-9781-efcb69394137\") " pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:28:42.666671 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.666645    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b502dcc5-5ffe-451f-9781-efcb69394137-model-cache\") pod \"simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d\" (UID: \"b502dcc5-5ffe-451f-9781-efcb69394137\") " pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:28:42.666945 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.666686    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt68l\" (UniqueName: \"kubernetes.io/projected/b502dcc5-5ffe-451f-9781-efcb69394137-kube-api-access-gt68l\") pod \"simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d\" (UID: \"b502dcc5-5ffe-451f-9781-efcb69394137\") " pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:28:42.666945 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.666714    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b502dcc5-5ffe-451f-9781-efcb69394137-home\") pod \"simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d\" (UID: \"b502dcc5-5ffe-451f-9781-efcb69394137\") " pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:28:42.666945 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.666768    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b502dcc5-5ffe-451f-9781-efcb69394137-dshm\") pod \"simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d\" (UID: \"b502dcc5-5ffe-451f-9781-efcb69394137\") " pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:28:42.667101 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.667055    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b502dcc5-5ffe-451f-9781-efcb69394137-tmp-dir\") pod \"simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d\" (UID: \"b502dcc5-5ffe-451f-9781-efcb69394137\") " pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:28:42.667101 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.667084    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b502dcc5-5ffe-451f-9781-efcb69394137-model-cache\") pod \"simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d\" (UID: \"b502dcc5-5ffe-451f-9781-efcb69394137\") " pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:28:42.667234 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.667109    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b502dcc5-5ffe-451f-9781-efcb69394137-home\") pod \"simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d\" (UID: \"b502dcc5-5ffe-451f-9781-efcb69394137\") " pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:28:42.668972 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.668948    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b502dcc5-5ffe-451f-9781-efcb69394137-dshm\") pod \"simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d\" (UID: \"b502dcc5-5ffe-451f-9781-efcb69394137\") " pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:28:42.669287 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.669271    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b502dcc5-5ffe-451f-9781-efcb69394137-tls-certs\") pod \"simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d\" (UID: \"b502dcc5-5ffe-451f-9781-efcb69394137\") " pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:28:42.675942 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.675918    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt68l\" (UniqueName: \"kubernetes.io/projected/b502dcc5-5ffe-451f-9781-efcb69394137-kube-api-access-gt68l\") pod \"simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d\" (UID: \"b502dcc5-5ffe-451f-9781-efcb69394137\") " pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:28:42.695954 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.695935    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:28:42.968071 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:42.968039    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"]
Sep 11 12:28:42.969992 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:28:42.969969    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb502dcc5_5ffe_451f_9781_efcb69394137.slice/crio-7e27f9bd9464ddcf34d83a2b9a6472c88393043ce17402ba883f00562707f955 WatchSource:0}: Error finding container 7e27f9bd9464ddcf34d83a2b9a6472c88393043ce17402ba883f00562707f955: Status 404 returned error can't find the container with id 7e27f9bd9464ddcf34d83a2b9a6472c88393043ce17402ba883f00562707f955
Sep 11 12:28:43.523749 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:43.523712    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d" event={"ID":"b502dcc5-5ffe-451f-9781-efcb69394137","Type":"ContainerStarted","Data":"75e03e61aa9206fed9d049cf36cdef17ab23b58823978a54e4f6d8bc30183b55"}
Sep 11 12:28:43.524132 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:43.523757    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d" event={"ID":"b502dcc5-5ffe-451f-9781-efcb69394137","Type":"ContainerStarted","Data":"7e27f9bd9464ddcf34d83a2b9a6472c88393043ce17402ba883f00562707f955"}
Sep 11 12:28:43.544659 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:43.544610    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d" podStartSLOduration=1.5445966549999999 podStartE2EDuration="1.544596655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:43.543512237 +0000 UTC m=+855.106292000" watchObservedRunningTime="2026-09-11 12:28:43.544596655 +0000 UTC m=+855.107376451"
Sep 11 12:28:50.287174 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.287132    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh"]
Sep 11 12:28:50.287641 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.287390    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh" podUID="2b9c8f9b-7b7a-4cc9-acea-956654d6d732" containerName="payload-processing" containerID="cri-o://255bf4a8df31146bdfa14419b2b6f6d59ea927a88a91341b3246261e083cc058" gracePeriod=30
Sep 11 12:28:50.306468 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.306436    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl"]
Sep 11 12:28:50.306754 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.306684    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl" podUID="8b526270-8769-48c5-a0cd-a01aeadefd3d" containerName="payload-pre-processing" containerID="cri-o://2fb964aa302ad3c1adc53af21b654f8a5cb7942e4e7c549c7db64dd9bc87e81c" gracePeriod=30
Sep 11 12:28:50.546051 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.545995    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh"
Sep 11 12:28:50.560180 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.560152    2561 generic.go:356] "Generic (PLEG): container finished" podID="8b526270-8769-48c5-a0cd-a01aeadefd3d" containerID="2fb964aa302ad3c1adc53af21b654f8a5cb7942e4e7c549c7db64dd9bc87e81c" exitCode=0
Sep 11 12:28:50.560311 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.560227    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl" event={"ID":"8b526270-8769-48c5-a0cd-a01aeadefd3d","Type":"ContainerDied","Data":"2fb964aa302ad3c1adc53af21b654f8a5cb7942e4e7c549c7db64dd9bc87e81c"}
Sep 11 12:28:50.561620 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.561596    2561 generic.go:356] "Generic (PLEG): container finished" podID="2b9c8f9b-7b7a-4cc9-acea-956654d6d732" containerID="255bf4a8df31146bdfa14419b2b6f6d59ea927a88a91341b3246261e083cc058" exitCode=0
Sep 11 12:28:50.561720 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.561623    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh" event={"ID":"2b9c8f9b-7b7a-4cc9-acea-956654d6d732","Type":"ContainerDied","Data":"255bf4a8df31146bdfa14419b2b6f6d59ea927a88a91341b3246261e083cc058"}
Sep 11 12:28:50.561720 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.561639    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh"
Sep 11 12:28:50.561720 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.561683    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh" event={"ID":"2b9c8f9b-7b7a-4cc9-acea-956654d6d732","Type":"ContainerDied","Data":"4e039badff5cb30dd6d3099fa01a21b59f63a2aee1faff6bd2aac5c1202daefb"}
Sep 11 12:28:50.561720 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.561714    2561 scope.go:117] "RemoveContainer" containerID="255bf4a8df31146bdfa14419b2b6f6d59ea927a88a91341b3246261e083cc058"
Sep 11 12:28:50.579801 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.579769    2561 scope.go:117] "RemoveContainer" containerID="255bf4a8df31146bdfa14419b2b6f6d59ea927a88a91341b3246261e083cc058"
Sep 11 12:28:50.580116 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:28:50.580097    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"255bf4a8df31146bdfa14419b2b6f6d59ea927a88a91341b3246261e083cc058\": container with ID starting with 255bf4a8df31146bdfa14419b2b6f6d59ea927a88a91341b3246261e083cc058 not found: ID does not exist" containerID="255bf4a8df31146bdfa14419b2b6f6d59ea927a88a91341b3246261e083cc058"
Sep 11 12:28:50.580213 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.580125    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"255bf4a8df31146bdfa14419b2b6f6d59ea927a88a91341b3246261e083cc058"} err="failed to get container status \"255bf4a8df31146bdfa14419b2b6f6d59ea927a88a91341b3246261e083cc058\": rpc error: code = NotFound desc = could not find container \"255bf4a8df31146bdfa14419b2b6f6d59ea927a88a91341b3246261e083cc058\": container with ID starting with 255bf4a8df31146bdfa14419b2b6f6d59ea927a88a91341b3246261e083cc058 not found: ID does not exist"
Sep 11 12:28:50.581559 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.581509    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl"
Sep 11 12:28:50.636500 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.636473    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9c8f9b-7b7a-4cc9-acea-956654d6d732-plugins-config-volume\") pod \"2b9c8f9b-7b7a-4cc9-acea-956654d6d732\" (UID: \"2b9c8f9b-7b7a-4cc9-acea-956654d6d732\") "
Sep 11 12:28:50.636645 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.636518    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnggm\" (UniqueName: \"kubernetes.io/projected/2b9c8f9b-7b7a-4cc9-acea-956654d6d732-kube-api-access-jnggm\") pod \"2b9c8f9b-7b7a-4cc9-acea-956654d6d732\" (UID: \"2b9c8f9b-7b7a-4cc9-acea-956654d6d732\") "
Sep 11 12:28:50.636645 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.636579    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b526270-8769-48c5-a0cd-a01aeadefd3d-plugins-config-volume\") pod \"8b526270-8769-48c5-a0cd-a01aeadefd3d\" (UID: \"8b526270-8769-48c5-a0cd-a01aeadefd3d\") "
Sep 11 12:28:50.636645 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.636620    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6f5rx\" (UniqueName: \"kubernetes.io/projected/8b526270-8769-48c5-a0cd-a01aeadefd3d-kube-api-access-6f5rx\") pod \"8b526270-8769-48c5-a0cd-a01aeadefd3d\" (UID: \"8b526270-8769-48c5-a0cd-a01aeadefd3d\") "
Sep 11 12:28:50.636918 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.636894    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b9c8f9b-7b7a-4cc9-acea-956654d6d732-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2b9c8f9b-7b7a-4cc9-acea-956654d6d732" (UID: "2b9c8f9b-7b7a-4cc9-acea-956654d6d732"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:28:50.636989 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.636959    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b526270-8769-48c5-a0cd-a01aeadefd3d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b526270-8769-48c5-a0cd-a01aeadefd3d" (UID: "8b526270-8769-48c5-a0cd-a01aeadefd3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:28:50.638668 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.638642    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9c8f9b-7b7a-4cc9-acea-956654d6d732-kube-api-access-jnggm" (OuterVolumeSpecName: "kube-api-access-jnggm") pod "2b9c8f9b-7b7a-4cc9-acea-956654d6d732" (UID: "2b9c8f9b-7b7a-4cc9-acea-956654d6d732"). InnerVolumeSpecName "kube-api-access-jnggm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:28:50.638895 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.638875    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b526270-8769-48c5-a0cd-a01aeadefd3d-kube-api-access-6f5rx" (OuterVolumeSpecName: "kube-api-access-6f5rx") pod "8b526270-8769-48c5-a0cd-a01aeadefd3d" (UID: "8b526270-8769-48c5-a0cd-a01aeadefd3d"). InnerVolumeSpecName "kube-api-access-6f5rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:28:50.737790 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.737767    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b526270-8769-48c5-a0cd-a01aeadefd3d-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:28:50.737790 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.737789    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f5rx\" (UniqueName: \"kubernetes.io/projected/8b526270-8769-48c5-a0cd-a01aeadefd3d-kube-api-access-6f5rx\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:28:50.737951 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.737799    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9c8f9b-7b7a-4cc9-acea-956654d6d732-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:28:50.737951 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.737807    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnggm\" (UniqueName: \"kubernetes.io/projected/2b9c8f9b-7b7a-4cc9-acea-956654d6d732-kube-api-access-jnggm\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:28:50.883499 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.883466    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh"]
Sep 11 12:28:50.885345 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.885322    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e8d45b5-64cdc976dc-qvxnh"]
Sep 11 12:28:50.932852 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:50.932820    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b9c8f9b-7b7a-4cc9-acea-956654d6d732" path="/var/lib/kubelet/pods/2b9c8f9b-7b7a-4cc9-acea-956654d6d732/volumes"
Sep 11 12:28:50.947816 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:28:50.947796    2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b526270_8769_48c5_a0cd_a01aeadefd3d.slice/crio-8526190092eb6bfc10cb03bd774775674473e731d56ec7dd3435a7fd7d8098ad\": RecentStats: unable to find data in memory cache]"
Sep 11 12:28:51.569319 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:51.569238    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl"
Sep 11 12:28:51.569725 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:51.569228    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl" event={"ID":"8b526270-8769-48c5-a0cd-a01aeadefd3d","Type":"ContainerDied","Data":"8526190092eb6bfc10cb03bd774775674473e731d56ec7dd3435a7fd7d8098ad"}
Sep 11 12:28:51.569725 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:51.569378    2561 scope.go:117] "RemoveContainer" containerID="2fb964aa302ad3c1adc53af21b654f8a5cb7942e4e7c549c7db64dd9bc87e81c"
Sep 11 12:28:51.593342 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:51.593317    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl"]
Sep 11 12:28:51.602705 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:51.602680    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7e8d45b5-869c664df8-w6dwl"]
Sep 11 12:28:52.697042 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:52.697014    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:28:52.697466 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:52.697297    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:28:52.709514 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:52.709492    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:28:52.932644 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:52.932610    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b526270-8769-48c5-a0cd-a01aeadefd3d" path="/var/lib/kubelet/pods/8b526270-8769-48c5-a0cd-a01aeadefd3d/volumes"
Sep 11 12:28:53.590129 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:53.590100    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:28:54.355756 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:54.355724    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"]
Sep 11 12:28:54.356127 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:54.356115    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b9c8f9b-7b7a-4cc9-acea-956654d6d732" containerName="payload-processing"
Sep 11 12:28:54.356175 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:54.356128    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b9c8f9b-7b7a-4cc9-acea-956654d6d732" containerName="payload-processing"
Sep 11 12:28:54.356175 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:54.356143    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b526270-8769-48c5-a0cd-a01aeadefd3d" containerName="payload-pre-processing"
Sep 11 12:28:54.356175 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:54.356148    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b526270-8769-48c5-a0cd-a01aeadefd3d" containerName="payload-pre-processing"
Sep 11 12:28:54.356284 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:54.356230    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b526270-8769-48c5-a0cd-a01aeadefd3d" containerName="payload-pre-processing"
Sep 11 12:28:54.356284 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:54.356243    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b9c8f9b-7b7a-4cc9-acea-956654d6d732" containerName="payload-processing"
Sep 11 12:28:54.358967 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:54.358949    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"
Sep 11 12:28:54.359871 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:54.359843    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d4285c-50e5-4ff6-9167-160f27dcd23e-metrics-certs\") pod \"maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp\" (UID: \"46d4285c-50e5-4ff6-9167-160f27dcd23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"
Sep 11 12:28:54.359971 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:54.359891    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q9dg\" (UniqueName: \"kubernetes.io/projected/46d4285c-50e5-4ff6-9167-160f27dcd23e-kube-api-access-9q9dg\") pod \"maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp\" (UID: \"46d4285c-50e5-4ff6-9167-160f27dcd23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"
Sep 11 12:28:54.360012 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:54.359980    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d4285c-50e5-4ff6-9167-160f27dcd23e-maas-api-tls\") pod \"maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp\" (UID: \"46d4285c-50e5-4ff6-9167-160f27dcd23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"
Sep 11 12:28:54.360110 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:54.360091    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d4285c-50e5-4ff6-9167-160f27dcd23e-openshift-service-ca\") pod \"maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp\" (UID: \"46d4285c-50e5-4ff6-9167-160f27dcd23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"
Sep 11 12:28:54.361491 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:54.361471    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-cba59841-serving-cert\""
Sep 11 12:28:54.369870 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:54.369843    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"]
Sep 11 12:28:54.460557 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:54.460522    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d4285c-50e5-4ff6-9167-160f27dcd23e-openshift-service-ca\") pod \"maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp\" (UID: \"46d4285c-50e5-4ff6-9167-160f27dcd23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"
Sep 11 12:28:54.460720 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:54.460575    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d4285c-50e5-4ff6-9167-160f27dcd23e-metrics-certs\") pod \"maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp\" (UID: \"46d4285c-50e5-4ff6-9167-160f27dcd23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"
Sep 11 12:28:54.460720 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:54.460604    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q9dg\" (UniqueName: \"kubernetes.io/projected/46d4285c-50e5-4ff6-9167-160f27dcd23e-kube-api-access-9q9dg\") pod \"maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp\" (UID: \"46d4285c-50e5-4ff6-9167-160f27dcd23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"
Sep 11 12:28:54.460720 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:54.460648    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d4285c-50e5-4ff6-9167-160f27dcd23e-maas-api-tls\") pod \"maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp\" (UID: \"46d4285c-50e5-4ff6-9167-160f27dcd23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"
Sep 11 12:28:54.460844 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:28:54.460766    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-serving-cert: secret "maas-api-e2e-migrate-cba59841-serving-cert" not found
Sep 11 12:28:54.460844 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:28:54.460834    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46d4285c-50e5-4ff6-9167-160f27dcd23e-maas-api-tls podName:46d4285c-50e5-4ff6-9167-160f27dcd23e nodeName:}" failed. No retries permitted until 2026-09-11 12:28:54.960815549 +0000 UTC m=+866.523595291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/46d4285c-50e5-4ff6-9167-160f27dcd23e-maas-api-tls") pod "maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp" (UID: "46d4285c-50e5-4ff6-9167-160f27dcd23e") : secret "maas-api-e2e-migrate-cba59841-serving-cert" not found
Sep 11 12:28:54.461208 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:54.461164    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d4285c-50e5-4ff6-9167-160f27dcd23e-openshift-service-ca\") pod \"maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp\" (UID: \"46d4285c-50e5-4ff6-9167-160f27dcd23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"
Sep 11 12:28:54.463023 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:54.463004    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d4285c-50e5-4ff6-9167-160f27dcd23e-metrics-certs\") pod \"maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp\" (UID: \"46d4285c-50e5-4ff6-9167-160f27dcd23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"
Sep 11 12:28:54.472491 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:54.472463    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q9dg\" (UniqueName: \"kubernetes.io/projected/46d4285c-50e5-4ff6-9167-160f27dcd23e-kube-api-access-9q9dg\") pod \"maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp\" (UID: \"46d4285c-50e5-4ff6-9167-160f27dcd23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"
Sep 11 12:28:54.964774 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:54.964742    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d4285c-50e5-4ff6-9167-160f27dcd23e-maas-api-tls\") pod \"maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp\" (UID: \"46d4285c-50e5-4ff6-9167-160f27dcd23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"
Sep 11 12:28:54.967139 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:54.967112    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d4285c-50e5-4ff6-9167-160f27dcd23e-maas-api-tls\") pod \"maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp\" (UID: \"46d4285c-50e5-4ff6-9167-160f27dcd23e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"
Sep 11 12:28:54.971081 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:54.971063    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"
Sep 11 12:28:55.094417 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.094384    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk"]
Sep 11 12:28:55.097513 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.097489    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk"
Sep 11 12:28:55.101415 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.101387    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-cba59841\""
Sep 11 12:28:55.101529 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.101507    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-cba59841-dockercfg-q8nbn\""
Sep 11 12:28:55.110255 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.110233    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk"]
Sep 11 12:28:55.115163 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.115142    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn"]
Sep 11 12:28:55.117332 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.117317    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn"
Sep 11 12:28:55.128897 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.128869    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn"]
Sep 11 12:28:55.166673 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.166636    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570bfc73-6fd8-4253-8e6c-91a722bc0fd7-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn\" (UID: \"570bfc73-6fd8-4253-8e6c-91a722bc0fd7\") " pod="openshift-ingress/payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn"
Sep 11 12:28:55.166884 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.166864    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2x5s\" (UniqueName: \"kubernetes.io/projected/570bfc73-6fd8-4253-8e6c-91a722bc0fd7-kube-api-access-q2x5s\") pod \"payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn\" (UID: \"570bfc73-6fd8-4253-8e6c-91a722bc0fd7\") " pod="openshift-ingress/payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn"
Sep 11 12:28:55.241641 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.241606    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"]
Sep 11 12:28:55.243893 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:28:55.243870    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d4285c_50e5_4ff6_9167_160f27dcd23e.slice/crio-6cae68a3dd2445ecc16b9048b5d1c37d21b06fe3af962f46aba1ae053c5a11b6 WatchSource:0}: Error finding container 6cae68a3dd2445ecc16b9048b5d1c37d21b06fe3af962f46aba1ae053c5a11b6: Status 404 returned error can't find the container with id 6cae68a3dd2445ecc16b9048b5d1c37d21b06fe3af962f46aba1ae053c5a11b6
Sep 11 12:28:55.267584 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.267554    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv4gp\" (UniqueName: \"kubernetes.io/projected/fb3986ea-1852-4a90-9eb1-2203a2931e06-kube-api-access-tv4gp\") pod \"payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk\" (UID: \"fb3986ea-1852-4a90-9eb1-2203a2931e06\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk"
Sep 11 12:28:55.267678 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.267608    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2x5s\" (UniqueName: \"kubernetes.io/projected/570bfc73-6fd8-4253-8e6c-91a722bc0fd7-kube-api-access-q2x5s\") pod \"payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn\" (UID: \"570bfc73-6fd8-4253-8e6c-91a722bc0fd7\") " pod="openshift-ingress/payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn"
Sep 11 12:28:55.267730 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.267692    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb3986ea-1852-4a90-9eb1-2203a2931e06-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk\" (UID: \"fb3986ea-1852-4a90-9eb1-2203a2931e06\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk"
Sep 11 12:28:55.267768 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.267735    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570bfc73-6fd8-4253-8e6c-91a722bc0fd7-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn\" (UID: \"570bfc73-6fd8-4253-8e6c-91a722bc0fd7\") " pod="openshift-ingress/payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn"
Sep 11 12:28:55.268646 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.268628    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570bfc73-6fd8-4253-8e6c-91a722bc0fd7-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn\" (UID: \"570bfc73-6fd8-4253-8e6c-91a722bc0fd7\") " pod="openshift-ingress/payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn"
Sep 11 12:28:55.277008 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.276990    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2x5s\" (UniqueName: \"kubernetes.io/projected/570bfc73-6fd8-4253-8e6c-91a722bc0fd7-kube-api-access-q2x5s\") pod \"payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn\" (UID: \"570bfc73-6fd8-4253-8e6c-91a722bc0fd7\") " pod="openshift-ingress/payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn"
Sep 11 12:28:55.368533 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.368504    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb3986ea-1852-4a90-9eb1-2203a2931e06-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk\" (UID: \"fb3986ea-1852-4a90-9eb1-2203a2931e06\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk"
Sep 11 12:28:55.368841 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.368561    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv4gp\" (UniqueName: \"kubernetes.io/projected/fb3986ea-1852-4a90-9eb1-2203a2931e06-kube-api-access-tv4gp\") pod \"payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk\" (UID: \"fb3986ea-1852-4a90-9eb1-2203a2931e06\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk"
Sep 11 12:28:55.369169 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.369150    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb3986ea-1852-4a90-9eb1-2203a2931e06-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk\" (UID: \"fb3986ea-1852-4a90-9eb1-2203a2931e06\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk"
Sep 11 12:28:55.376168 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.376147    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv4gp\" (UniqueName: \"kubernetes.io/projected/fb3986ea-1852-4a90-9eb1-2203a2931e06-kube-api-access-tv4gp\") pod \"payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk\" (UID: \"fb3986ea-1852-4a90-9eb1-2203a2931e06\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk"
Sep 11 12:28:55.407697 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.407672    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk"
Sep 11 12:28:55.427221 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.427200    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn"
Sep 11 12:28:55.590061 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.590031    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp" event={"ID":"46d4285c-50e5-4ff6-9167-160f27dcd23e","Type":"ContainerStarted","Data":"6cae68a3dd2445ecc16b9048b5d1c37d21b06fe3af962f46aba1ae053c5a11b6"}
Sep 11 12:28:55.685616 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.685590    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk"]
Sep 11 12:28:55.687735 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:28:55.687707    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb3986ea_1852_4a90_9eb1_2203a2931e06.slice/crio-a9891b53bfe0844504e677c266c835f22f5c3d161bed04622daf0e7c45c04dfb WatchSource:0}: Error finding container a9891b53bfe0844504e677c266c835f22f5c3d161bed04622daf0e7c45c04dfb: Status 404 returned error can't find the container with id a9891b53bfe0844504e677c266c835f22f5c3d161bed04622daf0e7c45c04dfb
Sep 11 12:28:55.711621 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:55.711530    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn"]
Sep 11 12:28:55.721302 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:28:55.721184    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod570bfc73_6fd8_4253_8e6c_91a722bc0fd7.slice/crio-adb66c9b5a020ac50de0f5f54a4f1f04991ce8bae3d321e3b86d7dfb5a98dc8b WatchSource:0}: Error finding container adb66c9b5a020ac50de0f5f54a4f1f04991ce8bae3d321e3b86d7dfb5a98dc8b: Status 404 returned error can't find the container with id adb66c9b5a020ac50de0f5f54a4f1f04991ce8bae3d321e3b86d7dfb5a98dc8b
Sep 11 12:28:56.596169 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:56.596131    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn" event={"ID":"570bfc73-6fd8-4253-8e6c-91a722bc0fd7","Type":"ContainerStarted","Data":"04781e0439b4af111f563d6f1fc6e33a66950409fcadb15b6dfb23a603bb9bd8"}
Sep 11 12:28:56.596594 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:56.596208    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn" event={"ID":"570bfc73-6fd8-4253-8e6c-91a722bc0fd7","Type":"ContainerStarted","Data":"adb66c9b5a020ac50de0f5f54a4f1f04991ce8bae3d321e3b86d7dfb5a98dc8b"}
Sep 11 12:28:56.596594 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:56.596240    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn"
Sep 11 12:28:56.597703 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:56.597675    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp" event={"ID":"46d4285c-50e5-4ff6-9167-160f27dcd23e","Type":"ContainerStarted","Data":"66c78dc01a936a4f94c84ae063862056ed2527f8400523744d116f6378683cbe"}
Sep 11 12:28:56.597844 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:56.597732    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"
Sep 11 12:28:56.599204 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:56.599162    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk" event={"ID":"fb3986ea-1852-4a90-9eb1-2203a2931e06","Type":"ContainerStarted","Data":"ae0ab61906e016b21c4fbe1e91621bad788daf032495062ecc3c176b1acf0a1c"}
Sep 11 12:28:56.599302 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:56.599227    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk" event={"ID":"fb3986ea-1852-4a90-9eb1-2203a2931e06","Type":"ContainerStarted","Data":"a9891b53bfe0844504e677c266c835f22f5c3d161bed04622daf0e7c45c04dfb"}
Sep 11 12:28:56.599302 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:56.599261    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk"
Sep 11 12:28:56.618208 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:56.618141    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn" podStartSLOduration=1.618129391 podStartE2EDuration="1.618129391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:56.616671397 +0000 UTC m=+868.179451159" watchObservedRunningTime="2026-09-11 12:28:56.618129391 +0000 UTC m=+868.180909154"
Sep 11 12:28:56.639792 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:56.639752    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp" podStartSLOduration=2.320080148 podStartE2EDuration="2.639741148s" totalImagesPullingTime="319.661ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:56.637041765 +0000 UTC m=+868.199821527" watchObservedRunningTime="2026-09-11 12:28:56.639741148 +0000 UTC m=+868.202520968"
Sep 11 12:28:56.656985 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:56.656950    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk" podStartSLOduration=1.656938909 podStartE2EDuration="1.656938909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:56.656230857 +0000 UTC m=+868.219010618" watchObservedRunningTime="2026-09-11 12:28:56.656938909 +0000 UTC m=+868.219718672"
Sep 11 12:28:59.935758 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:28:59.935723    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:29:02.611083 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:02.611053    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"
Sep 11 12:29:07.607246 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:07.607214    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn"
Sep 11 12:29:07.607617 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:07.607299    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk"
Sep 11 12:29:12.082126 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.082091    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"]
Sep 11 12:29:12.087170 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.087148    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:29:12.089632 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.089606    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cb20c8\"/\"openshift-service-ca.crt\""
Sep 11 12:29:12.090553 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.090530    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cb20c8\"/\"simulator-w4-cb20c8-kserve-self-signed-certs\""
Sep 11 12:29:12.090642 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.090561    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cb20c8\"/\"default-dockercfg-v4trq\""
Sep 11 12:29:12.090642 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.090593    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cb20c8\"/\"kube-root-ca.crt\""
Sep 11 12:29:12.095354 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.095331    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"]
Sep 11 12:29:12.224305 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.224276    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-model-cache\") pod \"simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n\" (UID: \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\") " pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:29:12.224463 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.224330    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv9mk\" (UniqueName: \"kubernetes.io/projected/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-kube-api-access-mv9mk\") pod \"simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n\" (UID: \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\") " pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:29:12.224463 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.224385    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-home\") pod \"simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n\" (UID: \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\") " pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:29:12.224536 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.224463    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-tmp-dir\") pod \"simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n\" (UID: \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\") " pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:29:12.224536 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.224491    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-dshm\") pod \"simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n\" (UID: \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\") " pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:29:12.224536 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.224524    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-tls-certs\") pod \"simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n\" (UID: \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\") " pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:29:12.325531 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.325496    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv9mk\" (UniqueName: \"kubernetes.io/projected/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-kube-api-access-mv9mk\") pod \"simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n\" (UID: \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\") " pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:29:12.325531 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.325532    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-home\") pod \"simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n\" (UID: \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\") " pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:29:12.325733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.325561    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-tmp-dir\") pod \"simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n\" (UID: \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\") " pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:29:12.325733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.325580    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-dshm\") pod \"simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n\" (UID: \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\") " pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:29:12.325733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.325603    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-tls-certs\") pod \"simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n\" (UID: \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\") " pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:29:12.325733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.325644    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-model-cache\") pod \"simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n\" (UID: \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\") " pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:29:12.325953 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.325931    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-home\") pod \"simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n\" (UID: \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\") " pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:29:12.326017 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.325954    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-tmp-dir\") pod \"simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n\" (UID: \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\") " pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:29:12.326076 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.326011    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-model-cache\") pod \"simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n\" (UID: \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\") " pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:29:12.327894 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.327863    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-dshm\") pod \"simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n\" (UID: \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\") " pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:29:12.328041 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.328025    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-tls-certs\") pod \"simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n\" (UID: \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\") " pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:29:12.333940 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.333895    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv9mk\" (UniqueName: \"kubernetes.io/projected/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-kube-api-access-mv9mk\") pod \"simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n\" (UID: \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\") " pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:29:12.397296 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.397244    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:29:12.669418 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:12.669244    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"]
Sep 11 12:29:12.671989 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:29:12.671958    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f6731d4_62c8_4e59_a6be_acc6a9ae6082.slice/crio-e3a64035c3d35ef4adbd99c3f678ad94eba5400b0fa8b3357501df9de2a505cd WatchSource:0}: Error finding container e3a64035c3d35ef4adbd99c3f678ad94eba5400b0fa8b3357501df9de2a505cd: Status 404 returned error can't find the container with id e3a64035c3d35ef4adbd99c3f678ad94eba5400b0fa8b3357501df9de2a505cd
Sep 11 12:29:13.680836 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:13.680792    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n" event={"ID":"5f6731d4-62c8-4e59-a6be-acc6a9ae6082","Type":"ContainerStarted","Data":"efbcc68072ab74d2e00b15a8e42ccf0bc013341d468966807100f1b2a4e0cb1c"}
Sep 11 12:29:13.681210 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:13.680846    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n" event={"ID":"5f6731d4-62c8-4e59-a6be-acc6a9ae6082","Type":"ContainerStarted","Data":"e3a64035c3d35ef4adbd99c3f678ad94eba5400b0fa8b3357501df9de2a505cd"}
Sep 11 12:29:13.704267 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:13.704221    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n" podStartSLOduration=1.704206309 podStartE2EDuration="1.704206309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:29:13.702576402 +0000 UTC m=+885.265356166" watchObservedRunningTime="2026-09-11 12:29:13.704206309 +0000 UTC m=+885.266986068"
Sep 11 12:29:18.730204 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:18.730169    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:29:22.397616 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:22.397536    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:29:22.397616 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:22.397594    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:29:22.410519 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:22.410491    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:29:22.736546 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:22.736516    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:29:25.599759 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:25.599729    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"]
Sep 11 12:29:25.600250 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:25.599977    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp" podUID="46d4285c-50e5-4ff6-9167-160f27dcd23e" containerName="maas-api" containerID="cri-o://66c78dc01a936a4f94c84ae063862056ed2527f8400523744d116f6378683cbe" gracePeriod=30
Sep 11 12:29:25.652369 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:25.652338    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn"]
Sep 11 12:29:25.652609 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:25.652581    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn" podUID="570bfc73-6fd8-4253-8e6c-91a722bc0fd7" containerName="payload-processing" containerID="cri-o://04781e0439b4af111f563d6f1fc6e33a66950409fcadb15b6dfb23a603bb9bd8" gracePeriod=30
Sep 11 12:29:25.658093 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:25.658067    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk"]
Sep 11 12:29:25.658336 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:25.658294    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk" podUID="fb3986ea-1852-4a90-9eb1-2203a2931e06" containerName="payload-pre-processing" containerID="cri-o://ae0ab61906e016b21c4fbe1e91621bad788daf032495062ecc3c176b1acf0a1c" gracePeriod=30
Sep 11 12:29:25.740943 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:25.740915    2561 generic.go:356] "Generic (PLEG): container finished" podID="46d4285c-50e5-4ff6-9167-160f27dcd23e" containerID="66c78dc01a936a4f94c84ae063862056ed2527f8400523744d116f6378683cbe" exitCode=0
Sep 11 12:29:25.741044 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:25.740982    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp" event={"ID":"46d4285c-50e5-4ff6-9167-160f27dcd23e","Type":"ContainerDied","Data":"66c78dc01a936a4f94c84ae063862056ed2527f8400523744d116f6378683cbe"}
Sep 11 12:29:25.900229 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:25.900203    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn"
Sep 11 12:29:25.941447 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:25.941425    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"
Sep 11 12:29:25.948085 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:25.948068    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk"
Sep 11 12:29:26.050353 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.050323    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q9dg\" (UniqueName: \"kubernetes.io/projected/46d4285c-50e5-4ff6-9167-160f27dcd23e-kube-api-access-9q9dg\") pod \"46d4285c-50e5-4ff6-9167-160f27dcd23e\" (UID: \"46d4285c-50e5-4ff6-9167-160f27dcd23e\") "
Sep 11 12:29:26.050519 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.050374    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d4285c-50e5-4ff6-9167-160f27dcd23e-maas-api-tls\") pod \"46d4285c-50e5-4ff6-9167-160f27dcd23e\" (UID: \"46d4285c-50e5-4ff6-9167-160f27dcd23e\") "
Sep 11 12:29:26.050519 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.050397    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2x5s\" (UniqueName: \"kubernetes.io/projected/570bfc73-6fd8-4253-8e6c-91a722bc0fd7-kube-api-access-q2x5s\") pod \"570bfc73-6fd8-4253-8e6c-91a722bc0fd7\" (UID: \"570bfc73-6fd8-4253-8e6c-91a722bc0fd7\") "
Sep 11 12:29:26.050519 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.050434    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tv4gp\" (UniqueName: \"kubernetes.io/projected/fb3986ea-1852-4a90-9eb1-2203a2931e06-kube-api-access-tv4gp\") pod \"fb3986ea-1852-4a90-9eb1-2203a2931e06\" (UID: \"fb3986ea-1852-4a90-9eb1-2203a2931e06\") "
Sep 11 12:29:26.050519 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.050463    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb3986ea-1852-4a90-9eb1-2203a2931e06-plugins-config-volume\") pod \"fb3986ea-1852-4a90-9eb1-2203a2931e06\" (UID: \"fb3986ea-1852-4a90-9eb1-2203a2931e06\") "
Sep 11 12:29:26.050519 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.050506    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570bfc73-6fd8-4253-8e6c-91a722bc0fd7-plugins-config-volume\") pod \"570bfc73-6fd8-4253-8e6c-91a722bc0fd7\" (UID: \"570bfc73-6fd8-4253-8e6c-91a722bc0fd7\") "
Sep 11 12:29:26.050786 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.050536    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d4285c-50e5-4ff6-9167-160f27dcd23e-metrics-certs\") pod \"46d4285c-50e5-4ff6-9167-160f27dcd23e\" (UID: \"46d4285c-50e5-4ff6-9167-160f27dcd23e\") "
Sep 11 12:29:26.050786 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.050564    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d4285c-50e5-4ff6-9167-160f27dcd23e-openshift-service-ca\") pod \"46d4285c-50e5-4ff6-9167-160f27dcd23e\" (UID: \"46d4285c-50e5-4ff6-9167-160f27dcd23e\") "
Sep 11 12:29:26.050960 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.050928    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb3986ea-1852-4a90-9eb1-2203a2931e06-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fb3986ea-1852-4a90-9eb1-2203a2931e06" (UID: "fb3986ea-1852-4a90-9eb1-2203a2931e06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:29:26.051250 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.051220    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d4285c-50e5-4ff6-9167-160f27dcd23e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "46d4285c-50e5-4ff6-9167-160f27dcd23e" (UID: "46d4285c-50e5-4ff6-9167-160f27dcd23e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:29:26.051357 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.051257    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/570bfc73-6fd8-4253-8e6c-91a722bc0fd7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "570bfc73-6fd8-4253-8e6c-91a722bc0fd7" (UID: "570bfc73-6fd8-4253-8e6c-91a722bc0fd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:29:26.052598 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.052570    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570bfc73-6fd8-4253-8e6c-91a722bc0fd7-kube-api-access-q2x5s" (OuterVolumeSpecName: "kube-api-access-q2x5s") pod "570bfc73-6fd8-4253-8e6c-91a722bc0fd7" (UID: "570bfc73-6fd8-4253-8e6c-91a722bc0fd7"). InnerVolumeSpecName "kube-api-access-q2x5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:29:26.052704 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.052659    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d4285c-50e5-4ff6-9167-160f27dcd23e-kube-api-access-9q9dg" (OuterVolumeSpecName: "kube-api-access-9q9dg") pod "46d4285c-50e5-4ff6-9167-160f27dcd23e" (UID: "46d4285c-50e5-4ff6-9167-160f27dcd23e"). InnerVolumeSpecName "kube-api-access-9q9dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:29:26.052816 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.052796    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46d4285c-50e5-4ff6-9167-160f27dcd23e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "46d4285c-50e5-4ff6-9167-160f27dcd23e" (UID: "46d4285c-50e5-4ff6-9167-160f27dcd23e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:29:26.053019 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.052991    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46d4285c-50e5-4ff6-9167-160f27dcd23e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "46d4285c-50e5-4ff6-9167-160f27dcd23e" (UID: "46d4285c-50e5-4ff6-9167-160f27dcd23e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:29:26.053085 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.053044    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb3986ea-1852-4a90-9eb1-2203a2931e06-kube-api-access-tv4gp" (OuterVolumeSpecName: "kube-api-access-tv4gp") pod "fb3986ea-1852-4a90-9eb1-2203a2931e06" (UID: "fb3986ea-1852-4a90-9eb1-2203a2931e06"). InnerVolumeSpecName "kube-api-access-tv4gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:29:26.152099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.152024    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d4285c-50e5-4ff6-9167-160f27dcd23e-openshift-service-ca\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:29:26.152099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.152054    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q9dg\" (UniqueName: \"kubernetes.io/projected/46d4285c-50e5-4ff6-9167-160f27dcd23e-kube-api-access-9q9dg\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:29:26.152099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.152068    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d4285c-50e5-4ff6-9167-160f27dcd23e-maas-api-tls\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:29:26.152099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.152080    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2x5s\" (UniqueName: \"kubernetes.io/projected/570bfc73-6fd8-4253-8e6c-91a722bc0fd7-kube-api-access-q2x5s\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:29:26.152099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.152093    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv4gp\" (UniqueName: \"kubernetes.io/projected/fb3986ea-1852-4a90-9eb1-2203a2931e06-kube-api-access-tv4gp\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:29:26.152356 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.152105    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb3986ea-1852-4a90-9eb1-2203a2931e06-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:29:26.152356 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.152117    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/570bfc73-6fd8-4253-8e6c-91a722bc0fd7-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:29:26.152356 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.152129    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d4285c-50e5-4ff6-9167-160f27dcd23e-metrics-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:29:26.748033 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.747989    2561 generic.go:356] "Generic (PLEG): container finished" podID="570bfc73-6fd8-4253-8e6c-91a722bc0fd7" containerID="04781e0439b4af111f563d6f1fc6e33a66950409fcadb15b6dfb23a603bb9bd8" exitCode=0
Sep 11 12:29:26.748033 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.748035    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn"
Sep 11 12:29:26.748513 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.748070    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn" event={"ID":"570bfc73-6fd8-4253-8e6c-91a722bc0fd7","Type":"ContainerDied","Data":"04781e0439b4af111f563d6f1fc6e33a66950409fcadb15b6dfb23a603bb9bd8"}
Sep 11 12:29:26.748513 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.748126    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn" event={"ID":"570bfc73-6fd8-4253-8e6c-91a722bc0fd7","Type":"ContainerDied","Data":"adb66c9b5a020ac50de0f5f54a4f1f04991ce8bae3d321e3b86d7dfb5a98dc8b"}
Sep 11 12:29:26.748513 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.748160    2561 scope.go:117] "RemoveContainer" containerID="04781e0439b4af111f563d6f1fc6e33a66950409fcadb15b6dfb23a603bb9bd8"
Sep 11 12:29:26.749813 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.749552    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"
Sep 11 12:29:26.749813 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.749551    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp" event={"ID":"46d4285c-50e5-4ff6-9167-160f27dcd23e","Type":"ContainerDied","Data":"6cae68a3dd2445ecc16b9048b5d1c37d21b06fe3af962f46aba1ae053c5a11b6"}
Sep 11 12:29:26.750999 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.750975    2561 generic.go:356] "Generic (PLEG): container finished" podID="fb3986ea-1852-4a90-9eb1-2203a2931e06" containerID="ae0ab61906e016b21c4fbe1e91621bad788daf032495062ecc3c176b1acf0a1c" exitCode=0
Sep 11 12:29:26.751134 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.751021    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk"
Sep 11 12:29:26.751134 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.751073    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk" event={"ID":"fb3986ea-1852-4a90-9eb1-2203a2931e06","Type":"ContainerDied","Data":"ae0ab61906e016b21c4fbe1e91621bad788daf032495062ecc3c176b1acf0a1c"}
Sep 11 12:29:26.751257 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.751132    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk" event={"ID":"fb3986ea-1852-4a90-9eb1-2203a2931e06","Type":"ContainerDied","Data":"a9891b53bfe0844504e677c266c835f22f5c3d161bed04622daf0e7c45c04dfb"}
Sep 11 12:29:26.758777 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.758758    2561 scope.go:117] "RemoveContainer" containerID="04781e0439b4af111f563d6f1fc6e33a66950409fcadb15b6dfb23a603bb9bd8"
Sep 11 12:29:26.759137 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:29:26.759111    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04781e0439b4af111f563d6f1fc6e33a66950409fcadb15b6dfb23a603bb9bd8\": container with ID starting with 04781e0439b4af111f563d6f1fc6e33a66950409fcadb15b6dfb23a603bb9bd8 not found: ID does not exist" containerID="04781e0439b4af111f563d6f1fc6e33a66950409fcadb15b6dfb23a603bb9bd8"
Sep 11 12:29:26.759317 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.759286    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04781e0439b4af111f563d6f1fc6e33a66950409fcadb15b6dfb23a603bb9bd8"} err="failed to get container status \"04781e0439b4af111f563d6f1fc6e33a66950409fcadb15b6dfb23a603bb9bd8\": rpc error: code = NotFound desc = could not find container \"04781e0439b4af111f563d6f1fc6e33a66950409fcadb15b6dfb23a603bb9bd8\": container with ID starting with 04781e0439b4af111f563d6f1fc6e33a66950409fcadb15b6dfb23a603bb9bd8 not found: ID does not exist"
Sep 11 12:29:26.759414 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.759319    2561 scope.go:117] "RemoveContainer" containerID="66c78dc01a936a4f94c84ae063862056ed2527f8400523744d116f6378683cbe"
Sep 11 12:29:26.769738 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.769720    2561 scope.go:117] "RemoveContainer" containerID="ae0ab61906e016b21c4fbe1e91621bad788daf032495062ecc3c176b1acf0a1c"
Sep 11 12:29:26.777837 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.777817    2561 scope.go:117] "RemoveContainer" containerID="ae0ab61906e016b21c4fbe1e91621bad788daf032495062ecc3c176b1acf0a1c"
Sep 11 12:29:26.778090 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:29:26.778071    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0ab61906e016b21c4fbe1e91621bad788daf032495062ecc3c176b1acf0a1c\": container with ID starting with ae0ab61906e016b21c4fbe1e91621bad788daf032495062ecc3c176b1acf0a1c not found: ID does not exist" containerID="ae0ab61906e016b21c4fbe1e91621bad788daf032495062ecc3c176b1acf0a1c"
Sep 11 12:29:26.778166 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.778094    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0ab61906e016b21c4fbe1e91621bad788daf032495062ecc3c176b1acf0a1c"} err="failed to get container status \"ae0ab61906e016b21c4fbe1e91621bad788daf032495062ecc3c176b1acf0a1c\": rpc error: code = NotFound desc = could not find container \"ae0ab61906e016b21c4fbe1e91621bad788daf032495062ecc3c176b1acf0a1c\": container with ID starting with ae0ab61906e016b21c4fbe1e91621bad788daf032495062ecc3c176b1acf0a1c not found: ID does not exist"
Sep 11 12:29:26.786115 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.786091    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn"]
Sep 11 12:29:26.792381 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.792363    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cba59841-fd7b68c7d-5t5xn"]
Sep 11 12:29:26.807561 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.807522    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"]
Sep 11 12:29:26.809781 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.809762    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cba59841-86b9c8d6d6-tktnp"]
Sep 11 12:29:26.822976 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.822952    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk"]
Sep 11 12:29:26.825299 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.825280    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cba59841-6d66b6c78b-7c4hk"]
Sep 11 12:29:26.932567 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.932488    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46d4285c-50e5-4ff6-9167-160f27dcd23e" path="/var/lib/kubelet/pods/46d4285c-50e5-4ff6-9167-160f27dcd23e/volumes"
Sep 11 12:29:26.932951 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.932930    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="570bfc73-6fd8-4253-8e6c-91a722bc0fd7" path="/var/lib/kubelet/pods/570bfc73-6fd8-4253-8e6c-91a722bc0fd7/volumes"
Sep 11 12:29:26.933372 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:26.933354    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb3986ea-1852-4a90-9eb1-2203a2931e06" path="/var/lib/kubelet/pods/fb3986ea-1852-4a90-9eb1-2203a2931e06/volumes"
Sep 11 12:29:28.881801 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:28.881776    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcfm6_88402687-4e5b-411c-be7a-4cb955f90425/ovn-acl-logging/0.log"
Sep 11 12:29:28.884493 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:28.884475    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcfm6_88402687-4e5b-411c-be7a-4cb955f90425/ovn-acl-logging/0.log"
Sep 11 12:29:30.240994 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.240963    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"]
Sep 11 12:29:30.241375 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.241356    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb3986ea-1852-4a90-9eb1-2203a2931e06" containerName="payload-pre-processing"
Sep 11 12:29:30.241375 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.241370    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb3986ea-1852-4a90-9eb1-2203a2931e06" containerName="payload-pre-processing"
Sep 11 12:29:30.241444 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.241382    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="570bfc73-6fd8-4253-8e6c-91a722bc0fd7" containerName="payload-processing"
Sep 11 12:29:30.241444 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.241388    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="570bfc73-6fd8-4253-8e6c-91a722bc0fd7" containerName="payload-processing"
Sep 11 12:29:30.241444 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.241410    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46d4285c-50e5-4ff6-9167-160f27dcd23e" containerName="maas-api"
Sep 11 12:29:30.241444 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.241416    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="46d4285c-50e5-4ff6-9167-160f27dcd23e" containerName="maas-api"
Sep 11 12:29:30.241564 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.241468    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="46d4285c-50e5-4ff6-9167-160f27dcd23e" containerName="maas-api"
Sep 11 12:29:30.241564 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.241476    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="570bfc73-6fd8-4253-8e6c-91a722bc0fd7" containerName="payload-processing"
Sep 11 12:29:30.241564 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.241483    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb3986ea-1852-4a90-9eb1-2203a2931e06" containerName="payload-pre-processing"
Sep 11 12:29:30.246035 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.246017    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"
Sep 11 12:29:30.248470 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.248450    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-74075c51-serving-cert\""
Sep 11 12:29:30.256888 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.256861    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"]
Sep 11 12:29:30.390840 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.390803    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae3d478-7bbf-43b9-a15d-4000aa093790-openshift-service-ca\") pod \"maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp\" (UID: \"7ae3d478-7bbf-43b9-a15d-4000aa093790\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"
Sep 11 12:29:30.390840 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.390846    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae3d478-7bbf-43b9-a15d-4000aa093790-metrics-certs\") pod \"maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp\" (UID: \"7ae3d478-7bbf-43b9-a15d-4000aa093790\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"
Sep 11 12:29:30.391051 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.390869    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btjws\" (UniqueName: \"kubernetes.io/projected/7ae3d478-7bbf-43b9-a15d-4000aa093790-kube-api-access-btjws\") pod \"maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp\" (UID: \"7ae3d478-7bbf-43b9-a15d-4000aa093790\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"
Sep 11 12:29:30.391051 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.390954    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae3d478-7bbf-43b9-a15d-4000aa093790-maas-api-tls\") pod \"maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp\" (UID: \"7ae3d478-7bbf-43b9-a15d-4000aa093790\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"
Sep 11 12:29:30.491923 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.491844    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae3d478-7bbf-43b9-a15d-4000aa093790-openshift-service-ca\") pod \"maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp\" (UID: \"7ae3d478-7bbf-43b9-a15d-4000aa093790\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"
Sep 11 12:29:30.491923 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.491882    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae3d478-7bbf-43b9-a15d-4000aa093790-metrics-certs\") pod \"maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp\" (UID: \"7ae3d478-7bbf-43b9-a15d-4000aa093790\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"
Sep 11 12:29:30.491923 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.491902    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btjws\" (UniqueName: \"kubernetes.io/projected/7ae3d478-7bbf-43b9-a15d-4000aa093790-kube-api-access-btjws\") pod \"maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp\" (UID: \"7ae3d478-7bbf-43b9-a15d-4000aa093790\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"
Sep 11 12:29:30.492121 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.491926    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae3d478-7bbf-43b9-a15d-4000aa093790-maas-api-tls\") pod \"maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp\" (UID: \"7ae3d478-7bbf-43b9-a15d-4000aa093790\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"
Sep 11 12:29:30.492121 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:29:30.492028    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-serving-cert: secret "maas-api-e2e-ait-74075c51-serving-cert" not found
Sep 11 12:29:30.492121 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:29:30.492084    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ae3d478-7bbf-43b9-a15d-4000aa093790-maas-api-tls podName:7ae3d478-7bbf-43b9-a15d-4000aa093790 nodeName:}" failed. No retries permitted until 2026-09-11 12:29:30.992065596 +0000 UTC m=+902.554845339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7ae3d478-7bbf-43b9-a15d-4000aa093790-maas-api-tls") pod "maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp" (UID: "7ae3d478-7bbf-43b9-a15d-4000aa093790") : secret "maas-api-e2e-ait-74075c51-serving-cert" not found
Sep 11 12:29:30.492572 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.492550    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae3d478-7bbf-43b9-a15d-4000aa093790-openshift-service-ca\") pod \"maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp\" (UID: \"7ae3d478-7bbf-43b9-a15d-4000aa093790\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"
Sep 11 12:29:30.494298 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.494280    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae3d478-7bbf-43b9-a15d-4000aa093790-metrics-certs\") pod \"maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp\" (UID: \"7ae3d478-7bbf-43b9-a15d-4000aa093790\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"
Sep 11 12:29:30.503415 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.503394    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btjws\" (UniqueName: \"kubernetes.io/projected/7ae3d478-7bbf-43b9-a15d-4000aa093790-kube-api-access-btjws\") pod \"maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp\" (UID: \"7ae3d478-7bbf-43b9-a15d-4000aa093790\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"
Sep 11 12:29:30.996481 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.996449    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae3d478-7bbf-43b9-a15d-4000aa093790-maas-api-tls\") pod \"maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp\" (UID: \"7ae3d478-7bbf-43b9-a15d-4000aa093790\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"
Sep 11 12:29:30.998847 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:30.998829    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae3d478-7bbf-43b9-a15d-4000aa093790-maas-api-tls\") pod \"maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp\" (UID: \"7ae3d478-7bbf-43b9-a15d-4000aa093790\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"
Sep 11 12:29:31.028022 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:31.027997    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf"]
Sep 11 12:29:31.031772 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:31.031754    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf"
Sep 11 12:29:31.034479 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:31.034462    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-74075c51-dockercfg-b8kbz\""
Sep 11 12:29:31.034573 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:31.034519    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-74075c51\""
Sep 11 12:29:31.044516 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:31.044495    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf"]
Sep 11 12:29:31.157729 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:31.157694    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"
Sep 11 12:29:31.198162 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:31.198136    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b383f57-9e14-4b9a-8fc5-d47f9358214c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf\" (UID: \"5b383f57-9e14-4b9a-8fc5-d47f9358214c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf"
Sep 11 12:29:31.198319 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:31.198226    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n82wp\" (UniqueName: \"kubernetes.io/projected/5b383f57-9e14-4b9a-8fc5-d47f9358214c-kube-api-access-n82wp\") pod \"payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf\" (UID: \"5b383f57-9e14-4b9a-8fc5-d47f9358214c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf"
Sep 11 12:29:31.299237 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:31.299131    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b383f57-9e14-4b9a-8fc5-d47f9358214c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf\" (UID: \"5b383f57-9e14-4b9a-8fc5-d47f9358214c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf"
Sep 11 12:29:31.299614 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:31.299337    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n82wp\" (UniqueName: \"kubernetes.io/projected/5b383f57-9e14-4b9a-8fc5-d47f9358214c-kube-api-access-n82wp\") pod \"payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf\" (UID: \"5b383f57-9e14-4b9a-8fc5-d47f9358214c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf"
Sep 11 12:29:31.299859 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:31.299839    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b383f57-9e14-4b9a-8fc5-d47f9358214c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf\" (UID: \"5b383f57-9e14-4b9a-8fc5-d47f9358214c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf"
Sep 11 12:29:31.320090 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:31.320065    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n82wp\" (UniqueName: \"kubernetes.io/projected/5b383f57-9e14-4b9a-8fc5-d47f9358214c-kube-api-access-n82wp\") pod \"payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf\" (UID: \"5b383f57-9e14-4b9a-8fc5-d47f9358214c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf"
Sep 11 12:29:31.341813 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:31.341787    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf"
Sep 11 12:29:31.434976 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:29:31.434943    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae3d478_7bbf_43b9_a15d_4000aa093790.slice/crio-55d9b38677b09a0bd2c903abdb055b7815856e54730ffd368b612c5717724b40 WatchSource:0}: Error finding container 55d9b38677b09a0bd2c903abdb055b7815856e54730ffd368b612c5717724b40: Status 404 returned error can't find the container with id 55d9b38677b09a0bd2c903abdb055b7815856e54730ffd368b612c5717724b40
Sep 11 12:29:31.435118 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:31.434981    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"]
Sep 11 12:29:31.614363 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:31.614335    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf"]
Sep 11 12:29:31.615316 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:29:31.615287    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b383f57_9e14_4b9a_8fc5_d47f9358214c.slice/crio-0448db4bea0785a35657ed0c5ce1bde17af8b2c2ee31eeb0b20b072fbee3c6af WatchSource:0}: Error finding container 0448db4bea0785a35657ed0c5ce1bde17af8b2c2ee31eeb0b20b072fbee3c6af: Status 404 returned error can't find the container with id 0448db4bea0785a35657ed0c5ce1bde17af8b2c2ee31eeb0b20b072fbee3c6af
Sep 11 12:29:31.734431 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:31.734393    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:29:31.780289 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:31.780242    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf" event={"ID":"5b383f57-9e14-4b9a-8fc5-d47f9358214c","Type":"ContainerStarted","Data":"412b894e8c421e0eb0a6730e6d5900a37e5478cd31fe8224f719d4889f0867ce"}
Sep 11 12:29:31.780456 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:31.780399    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf"
Sep 11 12:29:31.780456 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:31.780444    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf" event={"ID":"5b383f57-9e14-4b9a-8fc5-d47f9358214c","Type":"ContainerStarted","Data":"0448db4bea0785a35657ed0c5ce1bde17af8b2c2ee31eeb0b20b072fbee3c6af"}
Sep 11 12:29:31.781696 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:31.781673    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp" event={"ID":"7ae3d478-7bbf-43b9-a15d-4000aa093790","Type":"ContainerStarted","Data":"55d9b38677b09a0bd2c903abdb055b7815856e54730ffd368b612c5717724b40"}
Sep 11 12:29:31.797750 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:31.797712    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf" podStartSLOduration=0.797700497 podStartE2EDuration="797.700497ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:29:31.7968607 +0000 UTC m=+903.359640464" watchObservedRunningTime="2026-09-11 12:29:31.797700497 +0000 UTC m=+903.360480298"
Sep 11 12:29:32.683547 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.683511    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"]
Sep 11 12:29:32.687003 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.686987    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:29:32.689813 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.689786    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-9226c5\"/\"kube-root-ca.crt\""
Sep 11 12:29:32.689923 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.689813    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-9226c5\"/\"openshift-service-ca.crt\""
Sep 11 12:29:32.691091 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.691069    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-9226c5\"/\"default-dockercfg-c8q7j\""
Sep 11 12:29:32.691247 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.691107    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-9226c5\"/\"autores-model-9226c5-kserve-self-signed-certs\""
Sep 11 12:29:32.696029 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.696000    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"]
Sep 11 12:29:32.788088 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.788048    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp" event={"ID":"7ae3d478-7bbf-43b9-a15d-4000aa093790","Type":"ContainerStarted","Data":"9e068fe3bf5b764479a00df2d78b1cd33d78be95f1163d060dff3ab78de0723a"}
Sep 11 12:29:32.788268 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.788113    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"
Sep 11 12:29:32.806851 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.806805    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp" podStartSLOduration=2.433840436 podStartE2EDuration="2.806790288s" totalImagesPullingTime="372.949852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:29:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:29:32.806061853 +0000 UTC m=+904.368841622" watchObservedRunningTime="2026-09-11 12:29:32.806790288 +0000 UTC m=+904.369570099"
Sep 11 12:29:32.814716 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.814686    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7dmx\" (UniqueName: \"kubernetes.io/projected/598d756f-4bae-405e-8b48-307a71e60cb6-kube-api-access-n7dmx\") pod \"autores-model-9226c5-kserve-6d8788c9c5-lsllb\" (UID: \"598d756f-4bae-405e-8b48-307a71e60cb6\") " pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:29:32.814818 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.814735    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598d756f-4bae-405e-8b48-307a71e60cb6-tls-certs\") pod \"autores-model-9226c5-kserve-6d8788c9c5-lsllb\" (UID: \"598d756f-4bae-405e-8b48-307a71e60cb6\") " pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:29:32.814818 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.814806    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598d756f-4bae-405e-8b48-307a71e60cb6-home\") pod \"autores-model-9226c5-kserve-6d8788c9c5-lsllb\" (UID: \"598d756f-4bae-405e-8b48-307a71e60cb6\") " pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:29:32.814917 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.814879    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598d756f-4bae-405e-8b48-307a71e60cb6-model-cache\") pod \"autores-model-9226c5-kserve-6d8788c9c5-lsllb\" (UID: \"598d756f-4bae-405e-8b48-307a71e60cb6\") " pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:29:32.815073 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.814917    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598d756f-4bae-405e-8b48-307a71e60cb6-dshm\") pod \"autores-model-9226c5-kserve-6d8788c9c5-lsllb\" (UID: \"598d756f-4bae-405e-8b48-307a71e60cb6\") " pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:29:32.815073 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.814935    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598d756f-4bae-405e-8b48-307a71e60cb6-tmp-dir\") pod \"autores-model-9226c5-kserve-6d8788c9c5-lsllb\" (UID: \"598d756f-4bae-405e-8b48-307a71e60cb6\") " pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:29:32.916069 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.916030    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598d756f-4bae-405e-8b48-307a71e60cb6-model-cache\") pod \"autores-model-9226c5-kserve-6d8788c9c5-lsllb\" (UID: \"598d756f-4bae-405e-8b48-307a71e60cb6\") " pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:29:32.916298 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.916107    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598d756f-4bae-405e-8b48-307a71e60cb6-dshm\") pod \"autores-model-9226c5-kserve-6d8788c9c5-lsllb\" (UID: \"598d756f-4bae-405e-8b48-307a71e60cb6\") " pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:29:32.916298 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.916137    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598d756f-4bae-405e-8b48-307a71e60cb6-tmp-dir\") pod \"autores-model-9226c5-kserve-6d8788c9c5-lsllb\" (UID: \"598d756f-4bae-405e-8b48-307a71e60cb6\") " pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:29:32.916298 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.916174    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7dmx\" (UniqueName: \"kubernetes.io/projected/598d756f-4bae-405e-8b48-307a71e60cb6-kube-api-access-n7dmx\") pod \"autores-model-9226c5-kserve-6d8788c9c5-lsllb\" (UID: \"598d756f-4bae-405e-8b48-307a71e60cb6\") " pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:29:32.916298 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.916241    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598d756f-4bae-405e-8b48-307a71e60cb6-tls-certs\") pod \"autores-model-9226c5-kserve-6d8788c9c5-lsllb\" (UID: \"598d756f-4bae-405e-8b48-307a71e60cb6\") " pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:29:32.916524 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.916317    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598d756f-4bae-405e-8b48-307a71e60cb6-home\") pod \"autores-model-9226c5-kserve-6d8788c9c5-lsllb\" (UID: \"598d756f-4bae-405e-8b48-307a71e60cb6\") " pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:29:32.916572 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.916516    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598d756f-4bae-405e-8b48-307a71e60cb6-model-cache\") pod \"autores-model-9226c5-kserve-6d8788c9c5-lsllb\" (UID: \"598d756f-4bae-405e-8b48-307a71e60cb6\") " pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:29:32.916740 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.916713    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598d756f-4bae-405e-8b48-307a71e60cb6-tmp-dir\") pod \"autores-model-9226c5-kserve-6d8788c9c5-lsllb\" (UID: \"598d756f-4bae-405e-8b48-307a71e60cb6\") " pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:29:32.916863 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.916747    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598d756f-4bae-405e-8b48-307a71e60cb6-home\") pod \"autores-model-9226c5-kserve-6d8788c9c5-lsllb\" (UID: \"598d756f-4bae-405e-8b48-307a71e60cb6\") " pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:29:32.918672 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.918652    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598d756f-4bae-405e-8b48-307a71e60cb6-dshm\") pod \"autores-model-9226c5-kserve-6d8788c9c5-lsllb\" (UID: \"598d756f-4bae-405e-8b48-307a71e60cb6\") " pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:29:32.919027 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.919007    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598d756f-4bae-405e-8b48-307a71e60cb6-tls-certs\") pod \"autores-model-9226c5-kserve-6d8788c9c5-lsllb\" (UID: \"598d756f-4bae-405e-8b48-307a71e60cb6\") " pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:29:32.924886 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.924867    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7dmx\" (UniqueName: \"kubernetes.io/projected/598d756f-4bae-405e-8b48-307a71e60cb6-kube-api-access-n7dmx\") pod \"autores-model-9226c5-kserve-6d8788c9c5-lsllb\" (UID: \"598d756f-4bae-405e-8b48-307a71e60cb6\") " pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:29:32.997717 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:32.997684    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:29:33.272046 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:33.272020    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"]
Sep 11 12:29:33.273499 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:29:33.273465    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod598d756f_4bae_405e_8b48_307a71e60cb6.slice/crio-461b919fcfb5ae8c5eb16088e3092e59a8e6e37e98dff2cfcadd4fa83b09434a WatchSource:0}: Error finding container 461b919fcfb5ae8c5eb16088e3092e59a8e6e37e98dff2cfcadd4fa83b09434a: Status 404 returned error can't find the container with id 461b919fcfb5ae8c5eb16088e3092e59a8e6e37e98dff2cfcadd4fa83b09434a
Sep 11 12:29:33.794376 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:33.794333    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb" event={"ID":"598d756f-4bae-405e-8b48-307a71e60cb6","Type":"ContainerStarted","Data":"f7c1672dc52eaa4d88e72dc6bcd2827654e018e5641623c570029a6cb8c928d6"}
Sep 11 12:29:33.794748 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:33.794384    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb" event={"ID":"598d756f-4bae-405e-8b48-307a71e60cb6","Type":"ContainerStarted","Data":"461b919fcfb5ae8c5eb16088e3092e59a8e6e37e98dff2cfcadd4fa83b09434a"}
Sep 11 12:29:33.814122 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:33.814074    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb" podStartSLOduration=1.814059312 podStartE2EDuration="1.814059312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:29:33.812127395 +0000 UTC m=+905.374907172" watchObservedRunningTime="2026-09-11 12:29:33.814059312 +0000 UTC m=+905.376839117"
Sep 11 12:29:38.800460 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:38.800428    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"
Sep 11 12:29:42.789858 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:42.789829    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf"
Sep 11 12:29:42.997776 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:42.997745    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:29:42.997949 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:42.997790    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:29:43.010394 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:43.010371    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:29:43.853559 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:43.853531    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:29:50.128072 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:29:50.128037    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:30:00.143641 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:00.143598    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-74075c51-29818830-dbf9v"]
Sep 11 12:30:00.147155 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:00.147135    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-74075c51-29818830-dbf9v"
Sep 11 12:30:00.149462 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:00.149441    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\""
Sep 11 12:30:00.158844 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:00.158823    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-74075c51-29818830-dbf9v"
Sep 11 12:30:00.168878 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:00.168850    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-74075c51-29818830-dbf9v"]
Sep 11 12:30:00.441005 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:00.440965    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-74075c51-29818830-dbf9v"]
Sep 11 12:30:00.443216 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:30:00.443173    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430b1721_660c_4153_8b57_9105a9137f30.slice/crio-1778dcda7bd6247ffde7e29edb8bfe55d19383feb507c8f6e8955417c3339728 WatchSource:0}: Error finding container 1778dcda7bd6247ffde7e29edb8bfe55d19383feb507c8f6e8955417c3339728: Status 404 returned error can't find the container with id 1778dcda7bd6247ffde7e29edb8bfe55d19383feb507c8f6e8955417c3339728
Sep 11 12:30:00.769510 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:00.769476    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"]
Sep 11 12:30:00.769708 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:00.769682    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp" podUID="7ae3d478-7bbf-43b9-a15d-4000aa093790" containerName="maas-api" containerID="cri-o://9e068fe3bf5b764479a00df2d78b1cd33d78be95f1163d060dff3ab78de0723a" gracePeriod=30
Sep 11 12:30:00.803643 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:00.803620    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-74075c51-29818830-dbf9v"]
Sep 11 12:30:00.829089 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:00.829063    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf"]
Sep 11 12:30:00.830252 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:00.829350    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf" podUID="5b383f57-9e14-4b9a-8fc5-d47f9358214c" containerName="payload-pre-processing" containerID="cri-o://412b894e8c421e0eb0a6730e6d5900a37e5478cd31fe8224f719d4889f0867ce" gracePeriod=30
Sep 11 12:30:00.930512 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:00.924003    2561 generic.go:356] "Generic (PLEG): container finished" podID="7ae3d478-7bbf-43b9-a15d-4000aa093790" containerID="9e068fe3bf5b764479a00df2d78b1cd33d78be95f1163d060dff3ab78de0723a" exitCode=0
Sep 11 12:30:00.930512 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:00.924127    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp" event={"ID":"7ae3d478-7bbf-43b9-a15d-4000aa093790","Type":"ContainerDied","Data":"9e068fe3bf5b764479a00df2d78b1cd33d78be95f1163d060dff3ab78de0723a"}
Sep 11 12:30:00.937008 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:00.936984    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-74075c51-29818830-dbf9v" event={"ID":"430b1721-660c-4153-8b57-9105a9137f30","Type":"ContainerStarted","Data":"1778dcda7bd6247ffde7e29edb8bfe55d19383feb507c8f6e8955417c3339728"}
Sep 11 12:30:01.057709 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.057682    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"
Sep 11 12:30:01.087452 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.087424    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf"
Sep 11 12:30:01.168133 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.168103    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b383f57-9e14-4b9a-8fc5-d47f9358214c-plugins-config-volume\") pod \"5b383f57-9e14-4b9a-8fc5-d47f9358214c\" (UID: \"5b383f57-9e14-4b9a-8fc5-d47f9358214c\") "
Sep 11 12:30:01.168577 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.168146    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btjws\" (UniqueName: \"kubernetes.io/projected/7ae3d478-7bbf-43b9-a15d-4000aa093790-kube-api-access-btjws\") pod \"7ae3d478-7bbf-43b9-a15d-4000aa093790\" (UID: \"7ae3d478-7bbf-43b9-a15d-4000aa093790\") "
Sep 11 12:30:01.168577 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.168185    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n82wp\" (UniqueName: \"kubernetes.io/projected/5b383f57-9e14-4b9a-8fc5-d47f9358214c-kube-api-access-n82wp\") pod \"5b383f57-9e14-4b9a-8fc5-d47f9358214c\" (UID: \"5b383f57-9e14-4b9a-8fc5-d47f9358214c\") "
Sep 11 12:30:01.168577 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.168239    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae3d478-7bbf-43b9-a15d-4000aa093790-maas-api-tls\") pod \"7ae3d478-7bbf-43b9-a15d-4000aa093790\" (UID: \"7ae3d478-7bbf-43b9-a15d-4000aa093790\") "
Sep 11 12:30:01.168577 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.168278    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae3d478-7bbf-43b9-a15d-4000aa093790-openshift-service-ca\") pod \"7ae3d478-7bbf-43b9-a15d-4000aa093790\" (UID: \"7ae3d478-7bbf-43b9-a15d-4000aa093790\") "
Sep 11 12:30:01.168577 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.168317    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae3d478-7bbf-43b9-a15d-4000aa093790-metrics-certs\") pod \"7ae3d478-7bbf-43b9-a15d-4000aa093790\" (UID: \"7ae3d478-7bbf-43b9-a15d-4000aa093790\") "
Sep 11 12:30:01.168853 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.168642    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b383f57-9e14-4b9a-8fc5-d47f9358214c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5b383f57-9e14-4b9a-8fc5-d47f9358214c" (UID: "5b383f57-9e14-4b9a-8fc5-d47f9358214c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:30:01.168853 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.168691    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ae3d478-7bbf-43b9-a15d-4000aa093790-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7ae3d478-7bbf-43b9-a15d-4000aa093790" (UID: "7ae3d478-7bbf-43b9-a15d-4000aa093790"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:30:01.170523 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.170497    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae3d478-7bbf-43b9-a15d-4000aa093790-kube-api-access-btjws" (OuterVolumeSpecName: "kube-api-access-btjws") pod "7ae3d478-7bbf-43b9-a15d-4000aa093790" (UID: "7ae3d478-7bbf-43b9-a15d-4000aa093790"). InnerVolumeSpecName "kube-api-access-btjws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:30:01.170733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.170701    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ae3d478-7bbf-43b9-a15d-4000aa093790-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7ae3d478-7bbf-43b9-a15d-4000aa093790" (UID: "7ae3d478-7bbf-43b9-a15d-4000aa093790"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:30:01.170842 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.170777    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b383f57-9e14-4b9a-8fc5-d47f9358214c-kube-api-access-n82wp" (OuterVolumeSpecName: "kube-api-access-n82wp") pod "5b383f57-9e14-4b9a-8fc5-d47f9358214c" (UID: "5b383f57-9e14-4b9a-8fc5-d47f9358214c"). InnerVolumeSpecName "kube-api-access-n82wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:30:01.170842 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.170778    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ae3d478-7bbf-43b9-a15d-4000aa093790-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7ae3d478-7bbf-43b9-a15d-4000aa093790" (UID: "7ae3d478-7bbf-43b9-a15d-4000aa093790"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:30:01.269599 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.269559    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n82wp\" (UniqueName: \"kubernetes.io/projected/5b383f57-9e14-4b9a-8fc5-d47f9358214c-kube-api-access-n82wp\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:30:01.269599 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.269593    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ae3d478-7bbf-43b9-a15d-4000aa093790-maas-api-tls\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:30:01.269795 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.269606    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ae3d478-7bbf-43b9-a15d-4000aa093790-openshift-service-ca\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:30:01.269795 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.269618    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ae3d478-7bbf-43b9-a15d-4000aa093790-metrics-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:30:01.269795 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.269630    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b383f57-9e14-4b9a-8fc5-d47f9358214c-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:30:01.269795 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.269641    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btjws\" (UniqueName: \"kubernetes.io/projected/7ae3d478-7bbf-43b9-a15d-4000aa093790-kube-api-access-btjws\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:30:01.939247 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.939208    2561 generic.go:356] "Generic (PLEG): container finished" podID="5b383f57-9e14-4b9a-8fc5-d47f9358214c" containerID="412b894e8c421e0eb0a6730e6d5900a37e5478cd31fe8224f719d4889f0867ce" exitCode=0
Sep 11 12:30:01.939434 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.939260    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf"
Sep 11 12:30:01.939434 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.939362    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf" event={"ID":"5b383f57-9e14-4b9a-8fc5-d47f9358214c","Type":"ContainerDied","Data":"412b894e8c421e0eb0a6730e6d5900a37e5478cd31fe8224f719d4889f0867ce"}
Sep 11 12:30:01.939434 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.939424    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf" event={"ID":"5b383f57-9e14-4b9a-8fc5-d47f9358214c","Type":"ContainerDied","Data":"0448db4bea0785a35657ed0c5ce1bde17af8b2c2ee31eeb0b20b072fbee3c6af"}
Sep 11 12:30:01.939629 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.939450    2561 scope.go:117] "RemoveContainer" containerID="412b894e8c421e0eb0a6730e6d5900a37e5478cd31fe8224f719d4889f0867ce"
Sep 11 12:30:01.941200 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.941020    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp" event={"ID":"7ae3d478-7bbf-43b9-a15d-4000aa093790","Type":"ContainerDied","Data":"55d9b38677b09a0bd2c903abdb055b7815856e54730ffd368b612c5717724b40"}
Sep 11 12:30:01.941200 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.941061    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"
Sep 11 12:30:01.966249 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.966221    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"]
Sep 11 12:30:01.974777 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.974728    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-74075c51-59f6c4c6d8-2q4qp"]
Sep 11 12:30:01.987233 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.987205    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf"]
Sep 11 12:30:01.989231 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:01.989210    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-74075c51-5b85bf4958-v64mf"]
Sep 11 12:30:02.551096 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:02.551075    2561 scope.go:117] "RemoveContainer" containerID="412b894e8c421e0eb0a6730e6d5900a37e5478cd31fe8224f719d4889f0867ce"
Sep 11 12:30:02.551484 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:30:02.551462    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"412b894e8c421e0eb0a6730e6d5900a37e5478cd31fe8224f719d4889f0867ce\": container with ID starting with 412b894e8c421e0eb0a6730e6d5900a37e5478cd31fe8224f719d4889f0867ce not found: ID does not exist" containerID="412b894e8c421e0eb0a6730e6d5900a37e5478cd31fe8224f719d4889f0867ce"
Sep 11 12:30:02.551567 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:02.551499    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"412b894e8c421e0eb0a6730e6d5900a37e5478cd31fe8224f719d4889f0867ce"} err="failed to get container status \"412b894e8c421e0eb0a6730e6d5900a37e5478cd31fe8224f719d4889f0867ce\": rpc error: code = NotFound desc = could not find container \"412b894e8c421e0eb0a6730e6d5900a37e5478cd31fe8224f719d4889f0867ce\": container with ID starting with 412b894e8c421e0eb0a6730e6d5900a37e5478cd31fe8224f719d4889f0867ce not found: ID does not exist"
Sep 11 12:30:02.551567 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:02.551525    2561 scope.go:117] "RemoveContainer" containerID="9e068fe3bf5b764479a00df2d78b1cd33d78be95f1163d060dff3ab78de0723a"
Sep 11 12:30:02.932729 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:02.932697    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b383f57-9e14-4b9a-8fc5-d47f9358214c" path="/var/lib/kubelet/pods/5b383f57-9e14-4b9a-8fc5-d47f9358214c/volumes"
Sep 11 12:30:02.933162 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:02.933143    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ae3d478-7bbf-43b9-a15d-4000aa093790" path="/var/lib/kubelet/pods/7ae3d478-7bbf-43b9-a15d-4000aa093790/volumes"
Sep 11 12:30:02.948437 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:02.948410    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-74075c51-29818830-dbf9v" event={"ID":"430b1721-660c-4153-8b57-9105a9137f30","Type":"ContainerStarted","Data":"672d76d3d8aad93c7e264def82efda5b4c94b585cb26766ed5c52eae7e18bf73"}
Sep 11 12:30:02.948560 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:02.948439    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-74075c51-29818830-dbf9v" podUID="430b1721-660c-4153-8b57-9105a9137f30" containerName="cleanup" containerID="cri-o://672d76d3d8aad93c7e264def82efda5b4c94b585cb26766ed5c52eae7e18bf73" gracePeriod=30
Sep 11 12:30:02.963333 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:02.963288    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-74075c51-29818830-dbf9v" podStartSLOduration=0.79822239 podStartE2EDuration="2.963271469s" totalImagesPullingTime="2.165049079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:30:02.962888221 +0000 UTC m=+934.525667985" watchObservedRunningTime="2026-09-11 12:30:02.963271469 +0000 UTC m=+934.526051265"
Sep 11 12:30:03.402840 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:03.402813    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-74075c51-29818830-dbf9v"
Sep 11 12:30:03.955480 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:03.955446    2561 generic.go:356] "Generic (PLEG): container finished" podID="430b1721-660c-4153-8b57-9105a9137f30" containerID="672d76d3d8aad93c7e264def82efda5b4c94b585cb26766ed5c52eae7e18bf73" exitCode=6
Sep 11 12:30:03.955861 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:03.955489    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-74075c51-29818830-dbf9v"
Sep 11 12:30:03.955861 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:03.955533    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-74075c51-29818830-dbf9v" event={"ID":"430b1721-660c-4153-8b57-9105a9137f30","Type":"ContainerDied","Data":"672d76d3d8aad93c7e264def82efda5b4c94b585cb26766ed5c52eae7e18bf73"}
Sep 11 12:30:03.955861 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:03.955579    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-74075c51-29818830-dbf9v" event={"ID":"430b1721-660c-4153-8b57-9105a9137f30","Type":"ContainerDied","Data":"1778dcda7bd6247ffde7e29edb8bfe55d19383feb507c8f6e8955417c3339728"}
Sep 11 12:30:03.955861 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:03.955599    2561 scope.go:117] "RemoveContainer" containerID="672d76d3d8aad93c7e264def82efda5b4c94b585cb26766ed5c52eae7e18bf73"
Sep 11 12:30:03.964324 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:03.964309    2561 scope.go:117] "RemoveContainer" containerID="672d76d3d8aad93c7e264def82efda5b4c94b585cb26766ed5c52eae7e18bf73"
Sep 11 12:30:03.964563 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:30:03.964546    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"672d76d3d8aad93c7e264def82efda5b4c94b585cb26766ed5c52eae7e18bf73\": container with ID starting with 672d76d3d8aad93c7e264def82efda5b4c94b585cb26766ed5c52eae7e18bf73 not found: ID does not exist" containerID="672d76d3d8aad93c7e264def82efda5b4c94b585cb26766ed5c52eae7e18bf73"
Sep 11 12:30:03.964627 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:03.964569    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"672d76d3d8aad93c7e264def82efda5b4c94b585cb26766ed5c52eae7e18bf73"} err="failed to get container status \"672d76d3d8aad93c7e264def82efda5b4c94b585cb26766ed5c52eae7e18bf73\": rpc error: code = NotFound desc = could not find container \"672d76d3d8aad93c7e264def82efda5b4c94b585cb26766ed5c52eae7e18bf73\": container with ID starting with 672d76d3d8aad93c7e264def82efda5b4c94b585cb26766ed5c52eae7e18bf73 not found: ID does not exist"
Sep 11 12:30:03.977500 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:03.977478    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-74075c51-29818830-dbf9v"]
Sep 11 12:30:03.981167 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:03.981147    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-74075c51-29818830-dbf9v"]
Sep 11 12:30:04.932239 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:04.932207    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="430b1721-660c-4153-8b57-9105a9137f30" path="/var/lib/kubelet/pods/430b1721-660c-4153-8b57-9105a9137f30/volumes"
Sep 11 12:30:05.788024 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:05.787983    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"]
Sep 11 12:30:05.788574 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:05.788552    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="430b1721-660c-4153-8b57-9105a9137f30" containerName="cleanup"
Sep 11 12:30:05.788652 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:05.788576    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="430b1721-660c-4153-8b57-9105a9137f30" containerName="cleanup"
Sep 11 12:30:05.788652 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:05.788598    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ae3d478-7bbf-43b9-a15d-4000aa093790" containerName="maas-api"
Sep 11 12:30:05.788652 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:05.788608    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ae3d478-7bbf-43b9-a15d-4000aa093790" containerName="maas-api"
Sep 11 12:30:05.788652 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:05.788626    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b383f57-9e14-4b9a-8fc5-d47f9358214c" containerName="payload-pre-processing"
Sep 11 12:30:05.788652 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:05.788635    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b383f57-9e14-4b9a-8fc5-d47f9358214c" containerName="payload-pre-processing"
Sep 11 12:30:05.788896 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:05.788722    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="430b1721-660c-4153-8b57-9105a9137f30" containerName="cleanup"
Sep 11 12:30:05.788896 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:05.788736    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b383f57-9e14-4b9a-8fc5-d47f9358214c" containerName="payload-pre-processing"
Sep 11 12:30:05.788896 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:05.788747    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ae3d478-7bbf-43b9-a15d-4000aa093790" containerName="maas-api"
Sep 11 12:30:05.792994 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:05.792972    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:30:05.796596 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:05.796566    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-38a724\"/\"openshift-service-ca.crt\""
Sep 11 12:30:05.796596 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:05.796567    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-38a724\"/\"default-dockercfg-qv5kz\""
Sep 11 12:30:05.796737 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:05.796610    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-38a724\"/\"kube-root-ca.crt\""
Sep 11 12:30:05.796737 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:05.796610    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-38a724\"/\"test-model-38a724-kserve-self-signed-certs\""
Sep 11 12:30:05.801334 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:05.801316    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"]
Sep 11 12:30:05.914414 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:05.914385    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/402a229f-9035-408d-a896-f54fd8871ef1-home\") pod \"test-model-38a724-kserve-688b76459f-tzxwb\" (UID: \"402a229f-9035-408d-a896-f54fd8871ef1\") " pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:30:05.914593 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:05.914471    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/402a229f-9035-408d-a896-f54fd8871ef1-dshm\") pod \"test-model-38a724-kserve-688b76459f-tzxwb\" (UID: \"402a229f-9035-408d-a896-f54fd8871ef1\") " pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:30:05.914593 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:05.914507    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/402a229f-9035-408d-a896-f54fd8871ef1-tls-certs\") pod \"test-model-38a724-kserve-688b76459f-tzxwb\" (UID: \"402a229f-9035-408d-a896-f54fd8871ef1\") " pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:30:05.914593 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:05.914534    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/402a229f-9035-408d-a896-f54fd8871ef1-model-cache\") pod \"test-model-38a724-kserve-688b76459f-tzxwb\" (UID: \"402a229f-9035-408d-a896-f54fd8871ef1\") " pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:30:05.914593 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:05.914552    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/402a229f-9035-408d-a896-f54fd8871ef1-tmp-dir\") pod \"test-model-38a724-kserve-688b76459f-tzxwb\" (UID: \"402a229f-9035-408d-a896-f54fd8871ef1\") " pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:30:05.914593 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:05.914575    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t59b\" (UniqueName: \"kubernetes.io/projected/402a229f-9035-408d-a896-f54fd8871ef1-kube-api-access-7t59b\") pod \"test-model-38a724-kserve-688b76459f-tzxwb\" (UID: \"402a229f-9035-408d-a896-f54fd8871ef1\") " pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:30:06.015982 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:06.015947    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/402a229f-9035-408d-a896-f54fd8871ef1-home\") pod \"test-model-38a724-kserve-688b76459f-tzxwb\" (UID: \"402a229f-9035-408d-a896-f54fd8871ef1\") " pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:30:06.016145 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:06.016036    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/402a229f-9035-408d-a896-f54fd8871ef1-dshm\") pod \"test-model-38a724-kserve-688b76459f-tzxwb\" (UID: \"402a229f-9035-408d-a896-f54fd8871ef1\") " pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:30:06.016145 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:06.016083    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/402a229f-9035-408d-a896-f54fd8871ef1-tls-certs\") pod \"test-model-38a724-kserve-688b76459f-tzxwb\" (UID: \"402a229f-9035-408d-a896-f54fd8871ef1\") " pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:30:06.016145 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:06.016113    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/402a229f-9035-408d-a896-f54fd8871ef1-model-cache\") pod \"test-model-38a724-kserve-688b76459f-tzxwb\" (UID: \"402a229f-9035-408d-a896-f54fd8871ef1\") " pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:30:06.016145 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:06.016138    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/402a229f-9035-408d-a896-f54fd8871ef1-tmp-dir\") pod \"test-model-38a724-kserve-688b76459f-tzxwb\" (UID: \"402a229f-9035-408d-a896-f54fd8871ef1\") " pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:30:06.016406 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:06.016161    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t59b\" (UniqueName: \"kubernetes.io/projected/402a229f-9035-408d-a896-f54fd8871ef1-kube-api-access-7t59b\") pod \"test-model-38a724-kserve-688b76459f-tzxwb\" (UID: \"402a229f-9035-408d-a896-f54fd8871ef1\") " pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:30:06.016462 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:06.016415    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/402a229f-9035-408d-a896-f54fd8871ef1-home\") pod \"test-model-38a724-kserve-688b76459f-tzxwb\" (UID: \"402a229f-9035-408d-a896-f54fd8871ef1\") " pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:30:06.016512 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:06.016479    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/402a229f-9035-408d-a896-f54fd8871ef1-model-cache\") pod \"test-model-38a724-kserve-688b76459f-tzxwb\" (UID: \"402a229f-9035-408d-a896-f54fd8871ef1\") " pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:30:06.016627 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:06.016600    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/402a229f-9035-408d-a896-f54fd8871ef1-tmp-dir\") pod \"test-model-38a724-kserve-688b76459f-tzxwb\" (UID: \"402a229f-9035-408d-a896-f54fd8871ef1\") " pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:30:06.018382 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:06.018359    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/402a229f-9035-408d-a896-f54fd8871ef1-dshm\") pod \"test-model-38a724-kserve-688b76459f-tzxwb\" (UID: \"402a229f-9035-408d-a896-f54fd8871ef1\") " pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:30:06.018656 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:06.018636    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/402a229f-9035-408d-a896-f54fd8871ef1-tls-certs\") pod \"test-model-38a724-kserve-688b76459f-tzxwb\" (UID: \"402a229f-9035-408d-a896-f54fd8871ef1\") " pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:30:06.024100 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:06.024083    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t59b\" (UniqueName: \"kubernetes.io/projected/402a229f-9035-408d-a896-f54fd8871ef1-kube-api-access-7t59b\") pod \"test-model-38a724-kserve-688b76459f-tzxwb\" (UID: \"402a229f-9035-408d-a896-f54fd8871ef1\") " pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:30:06.103601 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:06.103534    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:30:06.373764 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:06.373739    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"]
Sep 11 12:30:06.376441 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:30:06.376415    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod402a229f_9035_408d_a896_f54fd8871ef1.slice/crio-7898f17800ef956d80b606568ef74c10d8820f234a1ffbdb836c459465856547 WatchSource:0}: Error finding container 7898f17800ef956d80b606568ef74c10d8820f234a1ffbdb836c459465856547: Status 404 returned error can't find the container with id 7898f17800ef956d80b606568ef74c10d8820f234a1ffbdb836c459465856547
Sep 11 12:30:06.971397 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:06.971366    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb" event={"ID":"402a229f-9035-408d-a896-f54fd8871ef1","Type":"ContainerStarted","Data":"f471b551b85da05471a1bb2aaafb6d7543fd5ceb8cca8d76b8aae261d2bc366b"}
Sep 11 12:30:06.971397 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:06.971403    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb" event={"ID":"402a229f-9035-408d-a896-f54fd8871ef1","Type":"ContainerStarted","Data":"7898f17800ef956d80b606568ef74c10d8820f234a1ffbdb836c459465856547"}
Sep 11 12:30:06.991649 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:06.991602    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb" podStartSLOduration=1.9915844539999998 podStartE2EDuration="1.991584454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:30:06.989607063 +0000 UTC m=+938.552386821" watchObservedRunningTime="2026-09-11 12:30:06.991584454 +0000 UTC m=+938.554364217"
Sep 11 12:30:07.304777 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.304734    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf"]
Sep 11 12:30:07.308868 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.308847    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf"
Sep 11 12:30:07.311337 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.311314    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-411110ed-serving-cert\""
Sep 11 12:30:07.315927 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.315904    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf"]
Sep 11 12:30:07.430945 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.430913    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5jmj\" (UniqueName: \"kubernetes.io/projected/fddb899a-621f-455d-af0d-cda361c9ca58-kube-api-access-m5jmj\") pod \"maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf\" (UID: \"fddb899a-621f-455d-af0d-cda361c9ca58\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf"
Sep 11 12:30:07.431099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.430953    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fddb899a-621f-455d-af0d-cda361c9ca58-openshift-service-ca\") pod \"maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf\" (UID: \"fddb899a-621f-455d-af0d-cda361c9ca58\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf"
Sep 11 12:30:07.431099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.431025    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fddb899a-621f-455d-af0d-cda361c9ca58-maas-api-tls\") pod \"maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf\" (UID: \"fddb899a-621f-455d-af0d-cda361c9ca58\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf"
Sep 11 12:30:07.431099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.431078    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fddb899a-621f-455d-af0d-cda361c9ca58-metrics-certs\") pod \"maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf\" (UID: \"fddb899a-621f-455d-af0d-cda361c9ca58\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf"
Sep 11 12:30:07.531523 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.531491    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fddb899a-621f-455d-af0d-cda361c9ca58-metrics-certs\") pod \"maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf\" (UID: \"fddb899a-621f-455d-af0d-cda361c9ca58\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf"
Sep 11 12:30:07.531695 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.531558    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5jmj\" (UniqueName: \"kubernetes.io/projected/fddb899a-621f-455d-af0d-cda361c9ca58-kube-api-access-m5jmj\") pod \"maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf\" (UID: \"fddb899a-621f-455d-af0d-cda361c9ca58\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf"
Sep 11 12:30:07.531695 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.531597    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fddb899a-621f-455d-af0d-cda361c9ca58-openshift-service-ca\") pod \"maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf\" (UID: \"fddb899a-621f-455d-af0d-cda361c9ca58\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf"
Sep 11 12:30:07.531826 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.531698    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fddb899a-621f-455d-af0d-cda361c9ca58-maas-api-tls\") pod \"maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf\" (UID: \"fddb899a-621f-455d-af0d-cda361c9ca58\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf"
Sep 11 12:30:07.532417 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.532394    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fddb899a-621f-455d-af0d-cda361c9ca58-openshift-service-ca\") pod \"maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf\" (UID: \"fddb899a-621f-455d-af0d-cda361c9ca58\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf"
Sep 11 12:30:07.534150 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.534129    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fddb899a-621f-455d-af0d-cda361c9ca58-maas-api-tls\") pod \"maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf\" (UID: \"fddb899a-621f-455d-af0d-cda361c9ca58\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf"
Sep 11 12:30:07.534253 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.534157    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fddb899a-621f-455d-af0d-cda361c9ca58-metrics-certs\") pod \"maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf\" (UID: \"fddb899a-621f-455d-af0d-cda361c9ca58\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf"
Sep 11 12:30:07.544091 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.544065    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5jmj\" (UniqueName: \"kubernetes.io/projected/fddb899a-621f-455d-af0d-cda361c9ca58-kube-api-access-m5jmj\") pod \"maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf\" (UID: \"fddb899a-621f-455d-af0d-cda361c9ca58\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf"
Sep 11 12:30:07.621782 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.621731    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf"
Sep 11 12:30:07.844506 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.844478    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:30:07.896772 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.896725    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf"]
Sep 11 12:30:07.898547 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:30:07.898519    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfddb899a_621f_455d_af0d_cda361c9ca58.slice/crio-8f563da4fd4fad273f953f189f4989f16d694fc6b31d2f787cfc1441b64e412e WatchSource:0}: Error finding container 8f563da4fd4fad273f953f189f4989f16d694fc6b31d2f787cfc1441b64e412e: Status 404 returned error can't find the container with id 8f563da4fd4fad273f953f189f4989f16d694fc6b31d2f787cfc1441b64e412e
Sep 11 12:30:07.973975 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.973949    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss"]
Sep 11 12:30:07.979231 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.979207    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf" event={"ID":"fddb899a-621f-455d-af0d-cda361c9ca58","Type":"ContainerStarted","Data":"8f563da4fd4fad273f953f189f4989f16d694fc6b31d2f787cfc1441b64e412e"}
Sep 11 12:30:07.979352 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.979263    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss"
Sep 11 12:30:07.981982 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.981962    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-411110ed-dockercfg-fnxjh\""
Sep 11 12:30:07.982093 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.982002    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-411110ed\""
Sep 11 12:30:07.987568 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:07.987547    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss"]
Sep 11 12:30:08.137024 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:08.136989    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zghts\" (UniqueName: \"kubernetes.io/projected/791c1bfc-e6d6-47d2-9ce7-e30708b89db0-kube-api-access-zghts\") pod \"payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss\" (UID: \"791c1bfc-e6d6-47d2-9ce7-e30708b89db0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss"
Sep 11 12:30:08.137183 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:08.137047    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/791c1bfc-e6d6-47d2-9ce7-e30708b89db0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss\" (UID: \"791c1bfc-e6d6-47d2-9ce7-e30708b89db0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss"
Sep 11 12:30:08.237765 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:08.237741    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zghts\" (UniqueName: \"kubernetes.io/projected/791c1bfc-e6d6-47d2-9ce7-e30708b89db0-kube-api-access-zghts\") pod \"payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss\" (UID: \"791c1bfc-e6d6-47d2-9ce7-e30708b89db0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss"
Sep 11 12:30:08.237886 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:08.237784    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/791c1bfc-e6d6-47d2-9ce7-e30708b89db0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss\" (UID: \"791c1bfc-e6d6-47d2-9ce7-e30708b89db0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss"
Sep 11 12:30:08.238506 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:08.238482    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/791c1bfc-e6d6-47d2-9ce7-e30708b89db0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss\" (UID: \"791c1bfc-e6d6-47d2-9ce7-e30708b89db0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss"
Sep 11 12:30:08.245369 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:08.245346    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zghts\" (UniqueName: \"kubernetes.io/projected/791c1bfc-e6d6-47d2-9ce7-e30708b89db0-kube-api-access-zghts\") pod \"payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss\" (UID: \"791c1bfc-e6d6-47d2-9ce7-e30708b89db0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss"
Sep 11 12:30:08.291050 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:08.291023    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss"
Sep 11 12:30:08.568351 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:08.568323    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss"]
Sep 11 12:30:08.569707 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:30:08.569686    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791c1bfc_e6d6_47d2_9ce7_e30708b89db0.slice/crio-7f4751aceb8a9f32a9ce617190c7c3fc1def498ee81b31e7d887ea42357d64d6 WatchSource:0}: Error finding container 7f4751aceb8a9f32a9ce617190c7c3fc1def498ee81b31e7d887ea42357d64d6: Status 404 returned error can't find the container with id 7f4751aceb8a9f32a9ce617190c7c3fc1def498ee81b31e7d887ea42357d64d6
Sep 11 12:30:08.985119 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:08.985048    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss" event={"ID":"791c1bfc-e6d6-47d2-9ce7-e30708b89db0","Type":"ContainerStarted","Data":"97a2319c2d9f8f4df31ae4d0e52b6b078ea2a688b0f0f81606535fb4e6fa8e49"}
Sep 11 12:30:08.985119 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:08.985117    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss"
Sep 11 12:30:08.985720 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:08.985143    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss" event={"ID":"791c1bfc-e6d6-47d2-9ce7-e30708b89db0","Type":"ContainerStarted","Data":"7f4751aceb8a9f32a9ce617190c7c3fc1def498ee81b31e7d887ea42357d64d6"}
Sep 11 12:30:08.986933 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:08.986907    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf" event={"ID":"fddb899a-621f-455d-af0d-cda361c9ca58","Type":"ContainerStarted","Data":"e4320aa29e700adba40881d444f8ad7de81e003e8f63d5c7c60430cbecab543e"}
Sep 11 12:30:08.987067 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:08.987042    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf"
Sep 11 12:30:09.004987 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:09.004939    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss" podStartSLOduration=2.004920779 podStartE2EDuration="2.004920779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:30:09.001543956 +0000 UTC m=+940.564323720" watchObservedRunningTime="2026-09-11 12:30:09.004920779 +0000 UTC m=+940.567700544"
Sep 11 12:30:09.020402 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:09.020346    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf" podStartSLOduration=1.6223061159999999 podStartE2EDuration="2.020328185s" totalImagesPullingTime="398.022069ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:30:09.020056688 +0000 UTC m=+940.582836453" watchObservedRunningTime="2026-09-11 12:30:09.020328185 +0000 UTC m=+940.583107949"
Sep 11 12:30:14.997622 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:14.997589    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf"
Sep 11 12:30:16.104024 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:16.103996    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:30:16.104024 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:16.104035    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:30:16.117035 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:16.117013    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:30:17.036050 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:17.036019    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:30:17.193034 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:17.193001    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9"]
Sep 11 12:30:17.193483 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:17.193227    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9" podUID="53eb3c50-53be-4a66-8e89-b548806b50b4" containerName="maas-api" containerID="cri-o://82afa211f8c6f36fb79bd7eb7e29745063ce392ea57fcfb69611e9efa6afc81f" gracePeriod=30
Sep 11 12:30:17.441167 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:17.441143    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9"
Sep 11 12:30:17.522873 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:17.522847    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkwzz\" (UniqueName: \"kubernetes.io/projected/53eb3c50-53be-4a66-8e89-b548806b50b4-kube-api-access-hkwzz\") pod \"53eb3c50-53be-4a66-8e89-b548806b50b4\" (UID: \"53eb3c50-53be-4a66-8e89-b548806b50b4\") "
Sep 11 12:30:17.523007 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:17.522884    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53eb3c50-53be-4a66-8e89-b548806b50b4-maas-api-tls\") pod \"53eb3c50-53be-4a66-8e89-b548806b50b4\" (UID: \"53eb3c50-53be-4a66-8e89-b548806b50b4\") "
Sep 11 12:30:17.523007 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:17.522972    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53eb3c50-53be-4a66-8e89-b548806b50b4-metrics-certs\") pod \"53eb3c50-53be-4a66-8e89-b548806b50b4\" (UID: \"53eb3c50-53be-4a66-8e89-b548806b50b4\") "
Sep 11 12:30:17.523078 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:17.523013    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53eb3c50-53be-4a66-8e89-b548806b50b4-openshift-service-ca\") pod \"53eb3c50-53be-4a66-8e89-b548806b50b4\" (UID: \"53eb3c50-53be-4a66-8e89-b548806b50b4\") "
Sep 11 12:30:17.523436 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:17.523412    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53eb3c50-53be-4a66-8e89-b548806b50b4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "53eb3c50-53be-4a66-8e89-b548806b50b4" (UID: "53eb3c50-53be-4a66-8e89-b548806b50b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:30:17.525012 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:17.524982    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53eb3c50-53be-4a66-8e89-b548806b50b4-kube-api-access-hkwzz" (OuterVolumeSpecName: "kube-api-access-hkwzz") pod "53eb3c50-53be-4a66-8e89-b548806b50b4" (UID: "53eb3c50-53be-4a66-8e89-b548806b50b4"). InnerVolumeSpecName "kube-api-access-hkwzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:30:17.525105 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:17.525005    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53eb3c50-53be-4a66-8e89-b548806b50b4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "53eb3c50-53be-4a66-8e89-b548806b50b4" (UID: "53eb3c50-53be-4a66-8e89-b548806b50b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:30:17.525105 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:17.525040    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53eb3c50-53be-4a66-8e89-b548806b50b4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "53eb3c50-53be-4a66-8e89-b548806b50b4" (UID: "53eb3c50-53be-4a66-8e89-b548806b50b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:30:17.623974 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:17.623888    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkwzz\" (UniqueName: \"kubernetes.io/projected/53eb3c50-53be-4a66-8e89-b548806b50b4-kube-api-access-hkwzz\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:30:17.623974 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:17.623925    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53eb3c50-53be-4a66-8e89-b548806b50b4-maas-api-tls\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:30:17.623974 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:17.623934    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53eb3c50-53be-4a66-8e89-b548806b50b4-metrics-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:30:17.623974 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:17.623944    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53eb3c50-53be-4a66-8e89-b548806b50b4-openshift-service-ca\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:30:18.031238 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:18.031207    2561 generic.go:356] "Generic (PLEG): container finished" podID="53eb3c50-53be-4a66-8e89-b548806b50b4" containerID="82afa211f8c6f36fb79bd7eb7e29745063ce392ea57fcfb69611e9efa6afc81f" exitCode=0
Sep 11 12:30:18.031399 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:18.031267    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9"
Sep 11 12:30:18.031399 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:18.031285    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9" event={"ID":"53eb3c50-53be-4a66-8e89-b548806b50b4","Type":"ContainerDied","Data":"82afa211f8c6f36fb79bd7eb7e29745063ce392ea57fcfb69611e9efa6afc81f"}
Sep 11 12:30:18.031399 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:18.031340    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9" event={"ID":"53eb3c50-53be-4a66-8e89-b548806b50b4","Type":"ContainerDied","Data":"a4ec61466cd22461ac95cf5a11e562182de7154747b80251609a95663f3d65b2"}
Sep 11 12:30:18.031399 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:18.031362    2561 scope.go:117] "RemoveContainer" containerID="82afa211f8c6f36fb79bd7eb7e29745063ce392ea57fcfb69611e9efa6afc81f"
Sep 11 12:30:18.040648 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:18.040631    2561 scope.go:117] "RemoveContainer" containerID="82afa211f8c6f36fb79bd7eb7e29745063ce392ea57fcfb69611e9efa6afc81f"
Sep 11 12:30:18.040971 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:30:18.040947    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82afa211f8c6f36fb79bd7eb7e29745063ce392ea57fcfb69611e9efa6afc81f\": container with ID starting with 82afa211f8c6f36fb79bd7eb7e29745063ce392ea57fcfb69611e9efa6afc81f not found: ID does not exist" containerID="82afa211f8c6f36fb79bd7eb7e29745063ce392ea57fcfb69611e9efa6afc81f"
Sep 11 12:30:18.041064 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:18.040978    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82afa211f8c6f36fb79bd7eb7e29745063ce392ea57fcfb69611e9efa6afc81f"} err="failed to get container status \"82afa211f8c6f36fb79bd7eb7e29745063ce392ea57fcfb69611e9efa6afc81f\": rpc error: code = NotFound desc = could not find container \"82afa211f8c6f36fb79bd7eb7e29745063ce392ea57fcfb69611e9efa6afc81f\": container with ID starting with 82afa211f8c6f36fb79bd7eb7e29745063ce392ea57fcfb69611e9efa6afc81f not found: ID does not exist"
Sep 11 12:30:18.058405 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:18.058381    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9"]
Sep 11 12:30:18.061347 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:18.061324    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9226c5-788c69c596-qdkt9"]
Sep 11 12:30:18.933060 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:18.933021    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53eb3c50-53be-4a66-8e89-b548806b50b4" path="/var/lib/kubelet/pods/53eb3c50-53be-4a66-8e89-b548806b50b4/volumes"
Sep 11 12:30:19.994393 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:19.994363    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss"
Sep 11 12:30:22.835754 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:22.835719    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"]
Sep 11 12:30:28.230928 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:28.230893    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:30:33.594232 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.594199    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"]
Sep 11 12:30:33.594786 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.594744    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53eb3c50-53be-4a66-8e89-b548806b50b4" containerName="maas-api"
Sep 11 12:30:33.594786 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.594765    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="53eb3c50-53be-4a66-8e89-b548806b50b4" containerName="maas-api"
Sep 11 12:30:33.594906 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.594881    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="53eb3c50-53be-4a66-8e89-b548806b50b4" containerName="maas-api"
Sep 11 12:30:33.601700 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.601674    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:30:33.604907 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.604886    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1a8ee3\"/\"premium-w0-1a8ee3-kserve-self-signed-certs\""
Sep 11 12:30:33.629580 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.629554    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"]
Sep 11 12:30:33.667311 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.667280    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-dshm\") pod \"premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj\" (UID: \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\") " pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:30:33.667481 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.667330    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-tls-certs\") pod \"premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj\" (UID: \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\") " pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:30:33.667481 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.667366    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dm5t\" (UniqueName: \"kubernetes.io/projected/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-kube-api-access-6dm5t\") pod \"premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj\" (UID: \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\") " pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:30:33.667481 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.667439    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-model-cache\") pod \"premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj\" (UID: \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\") " pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:30:33.667652 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.667495    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-tmp-dir\") pod \"premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj\" (UID: \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\") " pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:30:33.667652 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.667546    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-home\") pod \"premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj\" (UID: \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\") " pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:30:33.768338 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.768305    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-dshm\") pod \"premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj\" (UID: \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\") " pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:30:33.768569 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.768358    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-tls-certs\") pod \"premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj\" (UID: \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\") " pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:30:33.768569 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.768394    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dm5t\" (UniqueName: \"kubernetes.io/projected/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-kube-api-access-6dm5t\") pod \"premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj\" (UID: \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\") " pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:30:33.768569 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.768420    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-model-cache\") pod \"premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj\" (UID: \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\") " pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:30:33.768569 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.768445    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-tmp-dir\") pod \"premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj\" (UID: \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\") " pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:30:33.768569 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.768463    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-home\") pod \"premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj\" (UID: \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\") " pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:30:33.768841 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.768809    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-home\") pod \"premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj\" (UID: \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\") " pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:30:33.768881 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.768842    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-tmp-dir\") pod \"premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj\" (UID: \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\") " pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:30:33.768920 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.768886    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-model-cache\") pod \"premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj\" (UID: \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\") " pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:30:33.770682 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.770663    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-dshm\") pod \"premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj\" (UID: \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\") " pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:30:33.770864 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.770847    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-tls-certs\") pod \"premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj\" (UID: \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\") " pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:30:33.794089 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.794064    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dm5t\" (UniqueName: \"kubernetes.io/projected/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-kube-api-access-6dm5t\") pod \"premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj\" (UID: \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\") " pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:30:33.915548 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:33.915475    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:30:34.204000 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:34.203974    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"]
Sep 11 12:30:34.206044 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:30:34.206017    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d5f9a4b_ce98_4594_ac66_2d26d6144e86.slice/crio-278d4dd9872f1c09b087052cff647f8b7dcb79d8f6eab489beaefd628ff8df67 WatchSource:0}: Error finding container 278d4dd9872f1c09b087052cff647f8b7dcb79d8f6eab489beaefd628ff8df67: Status 404 returned error can't find the container with id 278d4dd9872f1c09b087052cff647f8b7dcb79d8f6eab489beaefd628ff8df67
Sep 11 12:30:35.120068 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:35.120028    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj" event={"ID":"0d5f9a4b-ce98-4594-ac66-2d26d6144e86","Type":"ContainerStarted","Data":"3b63788c43df0038ebb1d18608140b9eb2e38f1d86b14ede8904951e779f821b"}
Sep 11 12:30:35.120440 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:35.120080    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj" event={"ID":"0d5f9a4b-ce98-4594-ac66-2d26d6144e86","Type":"ContainerStarted","Data":"278d4dd9872f1c09b087052cff647f8b7dcb79d8f6eab489beaefd628ff8df67"}
Sep 11 12:30:37.207481 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.202540    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj" podStartSLOduration=4.202514478 podStartE2EDuration="4.202514478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:30:35.142590053 +0000 UTC m=+966.705369815" watchObservedRunningTime="2026-09-11 12:30:37.202514478 +0000 UTC m=+968.765294242"
Sep 11 12:30:37.207481 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.206741    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf"]
Sep 11 12:30:37.207481 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.207065    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf" podUID="fddb899a-621f-455d-af0d-cda361c9ca58" containerName="maas-api" containerID="cri-o://e4320aa29e700adba40881d444f8ad7de81e003e8f63d5c7c60430cbecab543e" gracePeriod=30
Sep 11 12:30:37.275417 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.275384    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss"]
Sep 11 12:30:37.275809 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.275782    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss" podUID="791c1bfc-e6d6-47d2-9ce7-e30708b89db0" containerName="payload-pre-processing" containerID="cri-o://97a2319c2d9f8f4df31ae4d0e52b6b078ea2a688b0f0f81606535fb4e6fa8e49" gracePeriod=30
Sep 11 12:30:37.521773 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.521745    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf"
Sep 11 12:30:37.537956 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.537934    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss"
Sep 11 12:30:37.611477 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.611443    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fddb899a-621f-455d-af0d-cda361c9ca58-metrics-certs\") pod \"fddb899a-621f-455d-af0d-cda361c9ca58\" (UID: \"fddb899a-621f-455d-af0d-cda361c9ca58\") "
Sep 11 12:30:37.611615 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.611495    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zghts\" (UniqueName: \"kubernetes.io/projected/791c1bfc-e6d6-47d2-9ce7-e30708b89db0-kube-api-access-zghts\") pod \"791c1bfc-e6d6-47d2-9ce7-e30708b89db0\" (UID: \"791c1bfc-e6d6-47d2-9ce7-e30708b89db0\") "
Sep 11 12:30:37.611615 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.611587    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/791c1bfc-e6d6-47d2-9ce7-e30708b89db0-plugins-config-volume\") pod \"791c1bfc-e6d6-47d2-9ce7-e30708b89db0\" (UID: \"791c1bfc-e6d6-47d2-9ce7-e30708b89db0\") "
Sep 11 12:30:37.611707 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.611619    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5jmj\" (UniqueName: \"kubernetes.io/projected/fddb899a-621f-455d-af0d-cda361c9ca58-kube-api-access-m5jmj\") pod \"fddb899a-621f-455d-af0d-cda361c9ca58\" (UID: \"fddb899a-621f-455d-af0d-cda361c9ca58\") "
Sep 11 12:30:37.611707 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.611642    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fddb899a-621f-455d-af0d-cda361c9ca58-openshift-service-ca\") pod \"fddb899a-621f-455d-af0d-cda361c9ca58\" (UID: \"fddb899a-621f-455d-af0d-cda361c9ca58\") "
Sep 11 12:30:37.611707 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.611656    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fddb899a-621f-455d-af0d-cda361c9ca58-maas-api-tls\") pod \"fddb899a-621f-455d-af0d-cda361c9ca58\" (UID: \"fddb899a-621f-455d-af0d-cda361c9ca58\") "
Sep 11 12:30:37.612040 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.612010    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fddb899a-621f-455d-af0d-cda361c9ca58-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fddb899a-621f-455d-af0d-cda361c9ca58" (UID: "fddb899a-621f-455d-af0d-cda361c9ca58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:30:37.612135 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.612016    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/791c1bfc-e6d6-47d2-9ce7-e30708b89db0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "791c1bfc-e6d6-47d2-9ce7-e30708b89db0" (UID: "791c1bfc-e6d6-47d2-9ce7-e30708b89db0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:30:37.615212 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.615132    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fddb899a-621f-455d-af0d-cda361c9ca58-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fddb899a-621f-455d-af0d-cda361c9ca58" (UID: "fddb899a-621f-455d-af0d-cda361c9ca58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:30:37.615332 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.615271    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fddb899a-621f-455d-af0d-cda361c9ca58-kube-api-access-m5jmj" (OuterVolumeSpecName: "kube-api-access-m5jmj") pod "fddb899a-621f-455d-af0d-cda361c9ca58" (UID: "fddb899a-621f-455d-af0d-cda361c9ca58"). InnerVolumeSpecName "kube-api-access-m5jmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:30:37.615332 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.615306    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/791c1bfc-e6d6-47d2-9ce7-e30708b89db0-kube-api-access-zghts" (OuterVolumeSpecName: "kube-api-access-zghts") pod "791c1bfc-e6d6-47d2-9ce7-e30708b89db0" (UID: "791c1bfc-e6d6-47d2-9ce7-e30708b89db0"). InnerVolumeSpecName "kube-api-access-zghts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:30:37.615622 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.615597    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fddb899a-621f-455d-af0d-cda361c9ca58-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fddb899a-621f-455d-af0d-cda361c9ca58" (UID: "fddb899a-621f-455d-af0d-cda361c9ca58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:30:37.713446 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.713360    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5jmj\" (UniqueName: \"kubernetes.io/projected/fddb899a-621f-455d-af0d-cda361c9ca58-kube-api-access-m5jmj\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:30:37.713446 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.713392    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fddb899a-621f-455d-af0d-cda361c9ca58-openshift-service-ca\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:30:37.713446 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.713403    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fddb899a-621f-455d-af0d-cda361c9ca58-maas-api-tls\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:30:37.713446 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.713411    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fddb899a-621f-455d-af0d-cda361c9ca58-metrics-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:30:37.713446 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.713419    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zghts\" (UniqueName: \"kubernetes.io/projected/791c1bfc-e6d6-47d2-9ce7-e30708b89db0-kube-api-access-zghts\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:30:37.713446 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.713428    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/791c1bfc-e6d6-47d2-9ce7-e30708b89db0-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:30:37.855243 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:37.855166    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb" podUID="598d756f-4bae-405e-8b48-307a71e60cb6" containerName="main" containerID="cri-o://f7c1672dc52eaa4d88e72dc6bcd2827654e018e5641623c570029a6cb8c928d6" gracePeriod=45
Sep 11 12:30:38.106692 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.106668    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:30:38.137002 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.136975    2561 generic.go:356] "Generic (PLEG): container finished" podID="598d756f-4bae-405e-8b48-307a71e60cb6" containerID="f7c1672dc52eaa4d88e72dc6bcd2827654e018e5641623c570029a6cb8c928d6" exitCode=0
Sep 11 12:30:38.137149 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.137020    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"
Sep 11 12:30:38.137149 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.137062    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb" event={"ID":"598d756f-4bae-405e-8b48-307a71e60cb6","Type":"ContainerDied","Data":"f7c1672dc52eaa4d88e72dc6bcd2827654e018e5641623c570029a6cb8c928d6"}
Sep 11 12:30:38.137149 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.137103    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb" event={"ID":"598d756f-4bae-405e-8b48-307a71e60cb6","Type":"ContainerDied","Data":"461b919fcfb5ae8c5eb16088e3092e59a8e6e37e98dff2cfcadd4fa83b09434a"}
Sep 11 12:30:38.137149 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.137125    2561 scope.go:117] "RemoveContainer" containerID="f7c1672dc52eaa4d88e72dc6bcd2827654e018e5641623c570029a6cb8c928d6"
Sep 11 12:30:38.138524 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.138500    2561 generic.go:356] "Generic (PLEG): container finished" podID="fddb899a-621f-455d-af0d-cda361c9ca58" containerID="e4320aa29e700adba40881d444f8ad7de81e003e8f63d5c7c60430cbecab543e" exitCode=0
Sep 11 12:30:38.138629 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.138534    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf"
Sep 11 12:30:38.138629 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.138585    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf" event={"ID":"fddb899a-621f-455d-af0d-cda361c9ca58","Type":"ContainerDied","Data":"e4320aa29e700adba40881d444f8ad7de81e003e8f63d5c7c60430cbecab543e"}
Sep 11 12:30:38.138739 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.138629    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf" event={"ID":"fddb899a-621f-455d-af0d-cda361c9ca58","Type":"ContainerDied","Data":"8f563da4fd4fad273f953f189f4989f16d694fc6b31d2f787cfc1441b64e412e"}
Sep 11 12:30:38.140160 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.140134    2561 generic.go:356] "Generic (PLEG): container finished" podID="791c1bfc-e6d6-47d2-9ce7-e30708b89db0" containerID="97a2319c2d9f8f4df31ae4d0e52b6b078ea2a688b0f0f81606535fb4e6fa8e49" exitCode=0
Sep 11 12:30:38.140302 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.140175    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss"
Sep 11 12:30:38.140302 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.140230    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss" event={"ID":"791c1bfc-e6d6-47d2-9ce7-e30708b89db0","Type":"ContainerDied","Data":"97a2319c2d9f8f4df31ae4d0e52b6b078ea2a688b0f0f81606535fb4e6fa8e49"}
Sep 11 12:30:38.140302 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.140270    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss" event={"ID":"791c1bfc-e6d6-47d2-9ce7-e30708b89db0","Type":"ContainerDied","Data":"7f4751aceb8a9f32a9ce617190c7c3fc1def498ee81b31e7d887ea42357d64d6"}
Sep 11 12:30:38.147604 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.147588    2561 scope.go:117] "RemoveContainer" containerID="f7c1672dc52eaa4d88e72dc6bcd2827654e018e5641623c570029a6cb8c928d6"
Sep 11 12:30:38.147923 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:30:38.147901    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7c1672dc52eaa4d88e72dc6bcd2827654e018e5641623c570029a6cb8c928d6\": container with ID starting with f7c1672dc52eaa4d88e72dc6bcd2827654e018e5641623c570029a6cb8c928d6 not found: ID does not exist" containerID="f7c1672dc52eaa4d88e72dc6bcd2827654e018e5641623c570029a6cb8c928d6"
Sep 11 12:30:38.148009 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.147933    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7c1672dc52eaa4d88e72dc6bcd2827654e018e5641623c570029a6cb8c928d6"} err="failed to get container status \"f7c1672dc52eaa4d88e72dc6bcd2827654e018e5641623c570029a6cb8c928d6\": rpc error: code = NotFound desc = could not find container \"f7c1672dc52eaa4d88e72dc6bcd2827654e018e5641623c570029a6cb8c928d6\": container with ID starting with f7c1672dc52eaa4d88e72dc6bcd2827654e018e5641623c570029a6cb8c928d6 not found: ID does not exist"
Sep 11 12:30:38.148009 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.147957    2561 scope.go:117] "RemoveContainer" containerID="e4320aa29e700adba40881d444f8ad7de81e003e8f63d5c7c60430cbecab543e"
Sep 11 12:30:38.157235 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.157219    2561 scope.go:117] "RemoveContainer" containerID="e4320aa29e700adba40881d444f8ad7de81e003e8f63d5c7c60430cbecab543e"
Sep 11 12:30:38.157512 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:30:38.157492    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4320aa29e700adba40881d444f8ad7de81e003e8f63d5c7c60430cbecab543e\": container with ID starting with e4320aa29e700adba40881d444f8ad7de81e003e8f63d5c7c60430cbecab543e not found: ID does not exist" containerID="e4320aa29e700adba40881d444f8ad7de81e003e8f63d5c7c60430cbecab543e"
Sep 11 12:30:38.157561 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.157518    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4320aa29e700adba40881d444f8ad7de81e003e8f63d5c7c60430cbecab543e"} err="failed to get container status \"e4320aa29e700adba40881d444f8ad7de81e003e8f63d5c7c60430cbecab543e\": rpc error: code = NotFound desc = could not find container \"e4320aa29e700adba40881d444f8ad7de81e003e8f63d5c7c60430cbecab543e\": container with ID starting with e4320aa29e700adba40881d444f8ad7de81e003e8f63d5c7c60430cbecab543e not found: ID does not exist"
Sep 11 12:30:38.157561 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.157535    2561 scope.go:117] "RemoveContainer" containerID="97a2319c2d9f8f4df31ae4d0e52b6b078ea2a688b0f0f81606535fb4e6fa8e49"
Sep 11 12:30:38.164891 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.164872    2561 scope.go:117] "RemoveContainer" containerID="97a2319c2d9f8f4df31ae4d0e52b6b078ea2a688b0f0f81606535fb4e6fa8e49"
Sep 11 12:30:38.165180 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:30:38.165162    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97a2319c2d9f8f4df31ae4d0e52b6b078ea2a688b0f0f81606535fb4e6fa8e49\": container with ID starting with 97a2319c2d9f8f4df31ae4d0e52b6b078ea2a688b0f0f81606535fb4e6fa8e49 not found: ID does not exist" containerID="97a2319c2d9f8f4df31ae4d0e52b6b078ea2a688b0f0f81606535fb4e6fa8e49"
Sep 11 12:30:38.165283 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.165204    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97a2319c2d9f8f4df31ae4d0e52b6b078ea2a688b0f0f81606535fb4e6fa8e49"} err="failed to get container status \"97a2319c2d9f8f4df31ae4d0e52b6b078ea2a688b0f0f81606535fb4e6fa8e49\": rpc error: code = NotFound desc = could not find container \"97a2319c2d9f8f4df31ae4d0e52b6b078ea2a688b0f0f81606535fb4e6fa8e49\": container with ID starting with 97a2319c2d9f8f4df31ae4d0e52b6b078ea2a688b0f0f81606535fb4e6fa8e49 not found: ID does not exist"
Sep 11 12:30:38.167125 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.167106    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf"]
Sep 11 12:30:38.169004 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.168986    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-411110ed-7b54f495b8-kxbjf"]
Sep 11 12:30:38.180245 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.180221    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss"]
Sep 11 12:30:38.183562 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.183539    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-411110ed-7d9644cdb-284ss"]
Sep 11 12:30:38.217977 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.217957    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598d756f-4bae-405e-8b48-307a71e60cb6-model-cache\") pod \"598d756f-4bae-405e-8b48-307a71e60cb6\" (UID: \"598d756f-4bae-405e-8b48-307a71e60cb6\") "
Sep 11 12:30:38.218372 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.218011    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598d756f-4bae-405e-8b48-307a71e60cb6-home\") pod \"598d756f-4bae-405e-8b48-307a71e60cb6\" (UID: \"598d756f-4bae-405e-8b48-307a71e60cb6\") "
Sep 11 12:30:38.218372 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.218035    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7dmx\" (UniqueName: \"kubernetes.io/projected/598d756f-4bae-405e-8b48-307a71e60cb6-kube-api-access-n7dmx\") pod \"598d756f-4bae-405e-8b48-307a71e60cb6\" (UID: \"598d756f-4bae-405e-8b48-307a71e60cb6\") "
Sep 11 12:30:38.218372 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.218073    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598d756f-4bae-405e-8b48-307a71e60cb6-dshm\") pod \"598d756f-4bae-405e-8b48-307a71e60cb6\" (UID: \"598d756f-4bae-405e-8b48-307a71e60cb6\") "
Sep 11 12:30:38.218372 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.218105    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598d756f-4bae-405e-8b48-307a71e60cb6-tmp-dir\") pod \"598d756f-4bae-405e-8b48-307a71e60cb6\" (UID: \"598d756f-4bae-405e-8b48-307a71e60cb6\") "
Sep 11 12:30:38.218372 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.218215    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598d756f-4bae-405e-8b48-307a71e60cb6-tls-certs\") pod \"598d756f-4bae-405e-8b48-307a71e60cb6\" (UID: \"598d756f-4bae-405e-8b48-307a71e60cb6\") "
Sep 11 12:30:38.218372 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.218263    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598d756f-4bae-405e-8b48-307a71e60cb6-model-cache" (OuterVolumeSpecName: "model-cache") pod "598d756f-4bae-405e-8b48-307a71e60cb6" (UID: "598d756f-4bae-405e-8b48-307a71e60cb6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:30:38.218372 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.218294    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598d756f-4bae-405e-8b48-307a71e60cb6-home" (OuterVolumeSpecName: "home") pod "598d756f-4bae-405e-8b48-307a71e60cb6" (UID: "598d756f-4bae-405e-8b48-307a71e60cb6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:30:38.218661 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.218480    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598d756f-4bae-405e-8b48-307a71e60cb6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "598d756f-4bae-405e-8b48-307a71e60cb6" (UID: "598d756f-4bae-405e-8b48-307a71e60cb6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:30:38.218661 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.218595    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598d756f-4bae-405e-8b48-307a71e60cb6-model-cache\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:30:38.218661 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.218613    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598d756f-4bae-405e-8b48-307a71e60cb6-home\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:30:38.218661 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.218648    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598d756f-4bae-405e-8b48-307a71e60cb6-tmp-dir\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:30:38.220415 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.220392    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/598d756f-4bae-405e-8b48-307a71e60cb6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "598d756f-4bae-405e-8b48-307a71e60cb6" (UID: "598d756f-4bae-405e-8b48-307a71e60cb6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:30:38.220581 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.220560    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598d756f-4bae-405e-8b48-307a71e60cb6-dshm" (OuterVolumeSpecName: "dshm") pod "598d756f-4bae-405e-8b48-307a71e60cb6" (UID: "598d756f-4bae-405e-8b48-307a71e60cb6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:30:38.220669 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.220581    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/598d756f-4bae-405e-8b48-307a71e60cb6-kube-api-access-n7dmx" (OuterVolumeSpecName: "kube-api-access-n7dmx") pod "598d756f-4bae-405e-8b48-307a71e60cb6" (UID: "598d756f-4bae-405e-8b48-307a71e60cb6"). InnerVolumeSpecName "kube-api-access-n7dmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:30:38.319477 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.319452    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598d756f-4bae-405e-8b48-307a71e60cb6-tls-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:30:38.319477 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.319476    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7dmx\" (UniqueName: \"kubernetes.io/projected/598d756f-4bae-405e-8b48-307a71e60cb6-kube-api-access-n7dmx\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:30:38.319614 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.319487    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598d756f-4bae-405e-8b48-307a71e60cb6-dshm\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:30:38.459662 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.459637    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"]
Sep 11 12:30:38.463969 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.463947    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-9226c5/autores-model-9226c5-kserve-6d8788c9c5-lsllb"]
Sep 11 12:30:38.934442 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.934409    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="598d756f-4bae-405e-8b48-307a71e60cb6" path="/var/lib/kubelet/pods/598d756f-4bae-405e-8b48-307a71e60cb6/volumes"
Sep 11 12:30:38.934804 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.934791    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="791c1bfc-e6d6-47d2-9ce7-e30708b89db0" path="/var/lib/kubelet/pods/791c1bfc-e6d6-47d2-9ce7-e30708b89db0/volumes"
Sep 11 12:30:38.935138 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:38.935126    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fddb899a-621f-455d-af0d-cda361c9ca58" path="/var/lib/kubelet/pods/fddb899a-621f-455d-af0d-cda361c9ca58/volumes"
Sep 11 12:30:43.915829 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:43.915789    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:30:43.916213 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:43.915851    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:30:43.928547 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:43.928516    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:30:44.184038 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:44.183951    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:30:50.039159 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:30:50.039122    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:31:13.333205 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:13.333152    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn"]
Sep 11 12:31:13.333757 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:13.333731    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="598d756f-4bae-405e-8b48-307a71e60cb6" containerName="main"
Sep 11 12:31:13.333939 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:13.333916    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="598d756f-4bae-405e-8b48-307a71e60cb6" containerName="main"
Sep 11 12:31:13.334032 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:13.333946    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fddb899a-621f-455d-af0d-cda361c9ca58" containerName="maas-api"
Sep 11 12:31:13.334032 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:13.333956    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fddb899a-621f-455d-af0d-cda361c9ca58" containerName="maas-api"
Sep 11 12:31:13.334032 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:13.333999    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791c1bfc-e6d6-47d2-9ce7-e30708b89db0" containerName="payload-pre-processing"
Sep 11 12:31:13.334032 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:13.334008    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="791c1bfc-e6d6-47d2-9ce7-e30708b89db0" containerName="payload-pre-processing"
Sep 11 12:31:13.334180 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:13.334124    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="598d756f-4bae-405e-8b48-307a71e60cb6" containerName="main"
Sep 11 12:31:13.334180 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:13.334138    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fddb899a-621f-455d-af0d-cda361c9ca58" containerName="maas-api"
Sep 11 12:31:13.334180 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:13.334148    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="791c1bfc-e6d6-47d2-9ce7-e30708b89db0" containerName="payload-pre-processing"
Sep 11 12:31:13.337691 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:13.337673    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn"
Sep 11 12:31:13.341227 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:13.341169    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-82ead5\""
Sep 11 12:31:13.341591 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:13.341564    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-82ead5-dockercfg-6dqdn\""
Sep 11 12:31:13.342801 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:13.342779    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn"]
Sep 11 12:31:13.448569 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:13.448543    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ff6929-bdfa-4c6f-8ff0-819cc1376d45-plugins-config-volume\") pod \"payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn\" (UID: \"98ff6929-bdfa-4c6f-8ff0-819cc1376d45\") " pod="openshift-ingress/payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn"
Sep 11 12:31:13.448710 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:13.448591    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q75rj\" (UniqueName: \"kubernetes.io/projected/98ff6929-bdfa-4c6f-8ff0-819cc1376d45-kube-api-access-q75rj\") pod \"payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn\" (UID: \"98ff6929-bdfa-4c6f-8ff0-819cc1376d45\") " pod="openshift-ingress/payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn"
Sep 11 12:31:13.549660 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:13.549635    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ff6929-bdfa-4c6f-8ff0-819cc1376d45-plugins-config-volume\") pod \"payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn\" (UID: \"98ff6929-bdfa-4c6f-8ff0-819cc1376d45\") " pod="openshift-ingress/payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn"
Sep 11 12:31:13.549778 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:13.549687    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q75rj\" (UniqueName: \"kubernetes.io/projected/98ff6929-bdfa-4c6f-8ff0-819cc1376d45-kube-api-access-q75rj\") pod \"payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn\" (UID: \"98ff6929-bdfa-4c6f-8ff0-819cc1376d45\") " pod="openshift-ingress/payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn"
Sep 11 12:31:13.550422 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:13.550404    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ff6929-bdfa-4c6f-8ff0-819cc1376d45-plugins-config-volume\") pod \"payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn\" (UID: \"98ff6929-bdfa-4c6f-8ff0-819cc1376d45\") " pod="openshift-ingress/payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn"
Sep 11 12:31:13.557990 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:13.557963    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q75rj\" (UniqueName: \"kubernetes.io/projected/98ff6929-bdfa-4c6f-8ff0-819cc1376d45-kube-api-access-q75rj\") pod \"payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn\" (UID: \"98ff6929-bdfa-4c6f-8ff0-819cc1376d45\") " pod="openshift-ingress/payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn"
Sep 11 12:31:13.649611 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:13.649534    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn"
Sep 11 12:31:13.927578 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:13.927556    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn"]
Sep 11 12:31:13.930043 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:31:13.930016    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ff6929_bdfa_4c6f_8ff0_819cc1376d45.slice/crio-5411f030b561599f025917cba1c019ba87c6560459b037768cfab07963067925 WatchSource:0}: Error finding container 5411f030b561599f025917cba1c019ba87c6560459b037768cfab07963067925: Status 404 returned error can't find the container with id 5411f030b561599f025917cba1c019ba87c6560459b037768cfab07963067925
Sep 11 12:31:14.315923 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:14.315891    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn" event={"ID":"98ff6929-bdfa-4c6f-8ff0-819cc1376d45","Type":"ContainerStarted","Data":"5c3165867b1f8900214e20a9394116a92996cabf3f336ecb33e74f790317bf21"}
Sep 11 12:31:14.316103 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:14.315942    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn" event={"ID":"98ff6929-bdfa-4c6f-8ff0-819cc1376d45","Type":"ContainerStarted","Data":"5411f030b561599f025917cba1c019ba87c6560459b037768cfab07963067925"}
Sep 11 12:31:14.316103 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:14.315993    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn"
Sep 11 12:31:14.335974 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:14.335915    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn" podStartSLOduration=1.335895638 podStartE2EDuration="1.335895638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:31:14.333089064 +0000 UTC m=+1005.895868841" watchObservedRunningTime="2026-09-11 12:31:14.335895638 +0000 UTC m=+1005.898675405"
Sep 11 12:31:25.323672 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:25.323642    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn"
Sep 11 12:31:25.762328 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:25.762295    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:31:50.051261 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:50.051221    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:31:58.001542 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.001508    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"]
Sep 11 12:31:58.004362 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.004346    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:31:58.007279 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.007248    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-82ead5\"/\"openshift-service-ca.crt\""
Sep 11 12:31:58.007420 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.007388    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-82ead5\"/\"kube-root-ca.crt\""
Sep 11 12:31:58.008428 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.008391    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-82ead5\"/\"explicit-model-82ead5-kserve-self-signed-certs\""
Sep 11 12:31:58.008428 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.008428    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-82ead5\"/\"default-dockercfg-pwqhb\""
Sep 11 12:31:58.017298 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.017263    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"]
Sep 11 12:31:58.045612 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.045582    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-home\") pod \"explicit-model-82ead5-kserve-7c645d5b7f-6k99r\" (UID: \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\") " pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:31:58.045727 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.045635    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-tls-certs\") pod \"explicit-model-82ead5-kserve-7c645d5b7f-6k99r\" (UID: \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\") " pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:31:58.045727 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.045703    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-dshm\") pod \"explicit-model-82ead5-kserve-7c645d5b7f-6k99r\" (UID: \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\") " pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:31:58.045805 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.045729    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-model-cache\") pod \"explicit-model-82ead5-kserve-7c645d5b7f-6k99r\" (UID: \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\") " pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:31:58.045805 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.045755    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-tmp-dir\") pod \"explicit-model-82ead5-kserve-7c645d5b7f-6k99r\" (UID: \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\") " pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:31:58.045805 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.045789    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpfgp\" (UniqueName: \"kubernetes.io/projected/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-kube-api-access-qpfgp\") pod \"explicit-model-82ead5-kserve-7c645d5b7f-6k99r\" (UID: \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\") " pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:31:58.147100 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.147069    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-home\") pod \"explicit-model-82ead5-kserve-7c645d5b7f-6k99r\" (UID: \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\") " pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:31:58.147289 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.147112    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-tls-certs\") pod \"explicit-model-82ead5-kserve-7c645d5b7f-6k99r\" (UID: \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\") " pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:31:58.147289 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.147135    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-dshm\") pod \"explicit-model-82ead5-kserve-7c645d5b7f-6k99r\" (UID: \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\") " pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:31:58.147289 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.147233    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-model-cache\") pod \"explicit-model-82ead5-kserve-7c645d5b7f-6k99r\" (UID: \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\") " pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:31:58.147289 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.147271    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-tmp-dir\") pod \"explicit-model-82ead5-kserve-7c645d5b7f-6k99r\" (UID: \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\") " pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:31:58.147505 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.147312    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpfgp\" (UniqueName: \"kubernetes.io/projected/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-kube-api-access-qpfgp\") pod \"explicit-model-82ead5-kserve-7c645d5b7f-6k99r\" (UID: \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\") " pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:31:58.147571 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.147510    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-home\") pod \"explicit-model-82ead5-kserve-7c645d5b7f-6k99r\" (UID: \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\") " pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:31:58.147630 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.147582    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-tmp-dir\") pod \"explicit-model-82ead5-kserve-7c645d5b7f-6k99r\" (UID: \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\") " pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:31:58.147686 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.147650    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-model-cache\") pod \"explicit-model-82ead5-kserve-7c645d5b7f-6k99r\" (UID: \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\") " pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:31:58.149327 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.149303    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-dshm\") pod \"explicit-model-82ead5-kserve-7c645d5b7f-6k99r\" (UID: \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\") " pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:31:58.149594 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.149578    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-tls-certs\") pod \"explicit-model-82ead5-kserve-7c645d5b7f-6k99r\" (UID: \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\") " pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:31:58.158097 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.158075    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpfgp\" (UniqueName: \"kubernetes.io/projected/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-kube-api-access-qpfgp\") pod \"explicit-model-82ead5-kserve-7c645d5b7f-6k99r\" (UID: \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\") " pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:31:58.313828 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.313736    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:31:58.592582 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:58.592504    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"]
Sep 11 12:31:58.595733 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:31:58.595705    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a7d7bfe_b2cd_429f_a029_ca02ac169f76.slice/crio-93fdae37db2c3408b6815d5cd08c7cadb2e76e863d2dd059790d5a755d175c11 WatchSource:0}: Error finding container 93fdae37db2c3408b6815d5cd08c7cadb2e76e863d2dd059790d5a755d175c11: Status 404 returned error can't find the container with id 93fdae37db2c3408b6815d5cd08c7cadb2e76e863d2dd059790d5a755d175c11
Sep 11 12:31:59.531720 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:59.531682    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r" event={"ID":"0a7d7bfe-b2cd-429f-a029-ca02ac169f76","Type":"ContainerStarted","Data":"33bebf1bf0df0b2ce6e8f595a225c67350fcb5e13c4604de78602b41acb89cc3"}
Sep 11 12:31:59.531720 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:59.531727    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r" event={"ID":"0a7d7bfe-b2cd-429f-a029-ca02ac169f76","Type":"ContainerStarted","Data":"93fdae37db2c3408b6815d5cd08c7cadb2e76e863d2dd059790d5a755d175c11"}
Sep 11 12:31:59.558168 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:31:59.558108    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r" podStartSLOduration=2.558091902 podStartE2EDuration="2.558091902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:31:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:31:59.557790437 +0000 UTC m=+1051.120570201" watchObservedRunningTime="2026-09-11 12:31:59.558091902 +0000 UTC m=+1051.120871664"
Sep 11 12:32:08.313935 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:08.313891    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:32:08.314479 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:08.313955    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:32:08.326595 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:08.326564    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:32:08.583833 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:08.583757    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:32:10.651764 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:10.651727    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:32:20.687576 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:20.687545    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"]
Sep 11 12:32:23.489106 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:23.489025    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn"]
Sep 11 12:32:23.492757 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:23.492740    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn"
Sep 11 12:32:23.495496 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:23.495475    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ad618118\""
Sep 11 12:32:23.504620 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:23.504599    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn"]
Sep 11 12:32:23.539827 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:23.539804    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf"]
Sep 11 12:32:23.543397 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:23.543382    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf"
Sep 11 12:32:23.545881 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:23.545857    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ad618118-dockercfg-xfr6q\""
Sep 11 12:32:23.556430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:23.556411    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf"]
Sep 11 12:32:23.581075 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:23.581054    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b6d48e-adfc-461d-92a7-42b0857f4b10-plugins-config-volume\") pod \"payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf\" (UID: \"b8b6d48e-adfc-461d-92a7-42b0857f4b10\") " pod="openshift-ingress/payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf"
Sep 11 12:32:23.581209 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:23.581123    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d6w9\" (UniqueName: \"kubernetes.io/projected/14a89d6e-f0a1-4eb5-a801-ad2e64f282a0-kube-api-access-5d6w9\") pod \"payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn\" (UID: \"14a89d6e-f0a1-4eb5-a801-ad2e64f282a0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn"
Sep 11 12:32:23.581209 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:23.581149    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a89d6e-f0a1-4eb5-a801-ad2e64f282a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn\" (UID: \"14a89d6e-f0a1-4eb5-a801-ad2e64f282a0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn"
Sep 11 12:32:23.581347 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:23.581322    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xff78\" (UniqueName: \"kubernetes.io/projected/b8b6d48e-adfc-461d-92a7-42b0857f4b10-kube-api-access-xff78\") pod \"payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf\" (UID: \"b8b6d48e-adfc-461d-92a7-42b0857f4b10\") " pod="openshift-ingress/payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf"
Sep 11 12:32:23.681951 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:23.681918    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d6w9\" (UniqueName: \"kubernetes.io/projected/14a89d6e-f0a1-4eb5-a801-ad2e64f282a0-kube-api-access-5d6w9\") pod \"payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn\" (UID: \"14a89d6e-f0a1-4eb5-a801-ad2e64f282a0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn"
Sep 11 12:32:23.681951 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:23.681955    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a89d6e-f0a1-4eb5-a801-ad2e64f282a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn\" (UID: \"14a89d6e-f0a1-4eb5-a801-ad2e64f282a0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn"
Sep 11 12:32:23.682162 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:23.682036    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xff78\" (UniqueName: \"kubernetes.io/projected/b8b6d48e-adfc-461d-92a7-42b0857f4b10-kube-api-access-xff78\") pod \"payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf\" (UID: \"b8b6d48e-adfc-461d-92a7-42b0857f4b10\") " pod="openshift-ingress/payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf"
Sep 11 12:32:23.682162 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:23.682079    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b6d48e-adfc-461d-92a7-42b0857f4b10-plugins-config-volume\") pod \"payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf\" (UID: \"b8b6d48e-adfc-461d-92a7-42b0857f4b10\") " pod="openshift-ingress/payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf"
Sep 11 12:32:23.682776 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:23.682755    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a89d6e-f0a1-4eb5-a801-ad2e64f282a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn\" (UID: \"14a89d6e-f0a1-4eb5-a801-ad2e64f282a0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn"
Sep 11 12:32:23.682822 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:23.682776    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b6d48e-adfc-461d-92a7-42b0857f4b10-plugins-config-volume\") pod \"payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf\" (UID: \"b8b6d48e-adfc-461d-92a7-42b0857f4b10\") " pod="openshift-ingress/payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf"
Sep 11 12:32:23.690297 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:23.690270    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xff78\" (UniqueName: \"kubernetes.io/projected/b8b6d48e-adfc-461d-92a7-42b0857f4b10-kube-api-access-xff78\") pod \"payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf\" (UID: \"b8b6d48e-adfc-461d-92a7-42b0857f4b10\") " pod="openshift-ingress/payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf"
Sep 11 12:32:23.690403 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:23.690328    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d6w9\" (UniqueName: \"kubernetes.io/projected/14a89d6e-f0a1-4eb5-a801-ad2e64f282a0-kube-api-access-5d6w9\") pod \"payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn\" (UID: \"14a89d6e-f0a1-4eb5-a801-ad2e64f282a0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn"
Sep 11 12:32:23.803915 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:23.803853    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn"
Sep 11 12:32:23.853061 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:23.853033    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf"
Sep 11 12:32:24.100774 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:24.100749    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn"]
Sep 11 12:32:24.102720 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:32:24.102693    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a89d6e_f0a1_4eb5_a801_ad2e64f282a0.slice/crio-f2264b4178963fbdfbd31b92dc2bb11220bd01eed948167a4e5c28dbbc7648d7 WatchSource:0}: Error finding container f2264b4178963fbdfbd31b92dc2bb11220bd01eed948167a4e5c28dbbc7648d7: Status 404 returned error can't find the container with id f2264b4178963fbdfbd31b92dc2bb11220bd01eed948167a4e5c28dbbc7648d7
Sep 11 12:32:24.133097 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:24.133076    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf"]
Sep 11 12:32:24.139370 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:32:24.139345    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b6d48e_adfc_461d_92a7_42b0857f4b10.slice/crio-a30c960d911be7819c3292ba764cf325c9f556bccf5dc70855a54f0db8ef3dc7 WatchSource:0}: Error finding container a30c960d911be7819c3292ba764cf325c9f556bccf5dc70855a54f0db8ef3dc7: Status 404 returned error can't find the container with id a30c960d911be7819c3292ba764cf325c9f556bccf5dc70855a54f0db8ef3dc7
Sep 11 12:32:24.649102 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:24.649044    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf" event={"ID":"b8b6d48e-adfc-461d-92a7-42b0857f4b10","Type":"ContainerStarted","Data":"47ad0d52fd31ec1026a7bb7496cfc3976f9e0292cef51d0249a4c8f746cf7ad0"}
Sep 11 12:32:24.649592 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:24.649160    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf"
Sep 11 12:32:24.649592 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:24.649204    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf" event={"ID":"b8b6d48e-adfc-461d-92a7-42b0857f4b10","Type":"ContainerStarted","Data":"a30c960d911be7819c3292ba764cf325c9f556bccf5dc70855a54f0db8ef3dc7"}
Sep 11 12:32:24.650875 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:24.650849    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn" event={"ID":"14a89d6e-f0a1-4eb5-a801-ad2e64f282a0","Type":"ContainerStarted","Data":"3943e3d9fb90a3df448ddc06b141dcd38a3daa96883c21d88cb26afd0e9bff34"}
Sep 11 12:32:24.651001 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:24.650889    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn" event={"ID":"14a89d6e-f0a1-4eb5-a801-ad2e64f282a0","Type":"ContainerStarted","Data":"f2264b4178963fbdfbd31b92dc2bb11220bd01eed948167a4e5c28dbbc7648d7"}
Sep 11 12:32:24.651001 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:24.650913    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn"
Sep 11 12:32:24.667070 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:24.667026    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf" podStartSLOduration=1.667013237 podStartE2EDuration="1.667013237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:32:24.665299157 +0000 UTC m=+1076.228078918" watchObservedRunningTime="2026-09-11 12:32:24.667013237 +0000 UTC m=+1076.229792999"
Sep 11 12:32:24.683096 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:24.683038    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn" podStartSLOduration=1.683018427 podStartE2EDuration="1.683018427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:32:24.681096944 +0000 UTC m=+1076.243876709" watchObservedRunningTime="2026-09-11 12:32:24.683018427 +0000 UTC m=+1076.245798193"
Sep 11 12:32:31.581638 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:31.581601    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn"]
Sep 11 12:32:31.582249 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:31.581885    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn" podUID="98ff6929-bdfa-4c6f-8ff0-819cc1376d45" containerName="payload-processing" containerID="cri-o://5c3165867b1f8900214e20a9394116a92996cabf3f336ecb33e74f790317bf21" gracePeriod=30
Sep 11 12:32:31.843311 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:31.843253    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn"
Sep 11 12:32:31.871588 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:31.871559    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q75rj\" (UniqueName: \"kubernetes.io/projected/98ff6929-bdfa-4c6f-8ff0-819cc1376d45-kube-api-access-q75rj\") pod \"98ff6929-bdfa-4c6f-8ff0-819cc1376d45\" (UID: \"98ff6929-bdfa-4c6f-8ff0-819cc1376d45\") "
Sep 11 12:32:31.871719 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:31.871673    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ff6929-bdfa-4c6f-8ff0-819cc1376d45-plugins-config-volume\") pod \"98ff6929-bdfa-4c6f-8ff0-819cc1376d45\" (UID: \"98ff6929-bdfa-4c6f-8ff0-819cc1376d45\") "
Sep 11 12:32:31.872081 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:31.872057    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ff6929-bdfa-4c6f-8ff0-819cc1376d45-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "98ff6929-bdfa-4c6f-8ff0-819cc1376d45" (UID: "98ff6929-bdfa-4c6f-8ff0-819cc1376d45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:32:31.873660 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:31.873633    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ff6929-bdfa-4c6f-8ff0-819cc1376d45-kube-api-access-q75rj" (OuterVolumeSpecName: "kube-api-access-q75rj") pod "98ff6929-bdfa-4c6f-8ff0-819cc1376d45" (UID: "98ff6929-bdfa-4c6f-8ff0-819cc1376d45"). InnerVolumeSpecName "kube-api-access-q75rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:32:31.972572 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:31.972542    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q75rj\" (UniqueName: \"kubernetes.io/projected/98ff6929-bdfa-4c6f-8ff0-819cc1376d45-kube-api-access-q75rj\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:32:31.972572 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:31.972568    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98ff6929-bdfa-4c6f-8ff0-819cc1376d45-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:32:32.690872 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:32.690835    2561 generic.go:356] "Generic (PLEG): container finished" podID="98ff6929-bdfa-4c6f-8ff0-819cc1376d45" containerID="5c3165867b1f8900214e20a9394116a92996cabf3f336ecb33e74f790317bf21" exitCode=0
Sep 11 12:32:32.691323 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:32.690879    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn"
Sep 11 12:32:32.691323 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:32.690886    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn" event={"ID":"98ff6929-bdfa-4c6f-8ff0-819cc1376d45","Type":"ContainerDied","Data":"5c3165867b1f8900214e20a9394116a92996cabf3f336ecb33e74f790317bf21"}
Sep 11 12:32:32.691323 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:32.690931    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn" event={"ID":"98ff6929-bdfa-4c6f-8ff0-819cc1376d45","Type":"ContainerDied","Data":"5411f030b561599f025917cba1c019ba87c6560459b037768cfab07963067925"}
Sep 11 12:32:32.691323 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:32.690957    2561 scope.go:117] "RemoveContainer" containerID="5c3165867b1f8900214e20a9394116a92996cabf3f336ecb33e74f790317bf21"
Sep 11 12:32:32.700176 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:32.700154    2561 scope.go:117] "RemoveContainer" containerID="5c3165867b1f8900214e20a9394116a92996cabf3f336ecb33e74f790317bf21"
Sep 11 12:32:32.700493 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:32:32.700475    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c3165867b1f8900214e20a9394116a92996cabf3f336ecb33e74f790317bf21\": container with ID starting with 5c3165867b1f8900214e20a9394116a92996cabf3f336ecb33e74f790317bf21 not found: ID does not exist" containerID="5c3165867b1f8900214e20a9394116a92996cabf3f336ecb33e74f790317bf21"
Sep 11 12:32:32.700579 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:32.700500    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c3165867b1f8900214e20a9394116a92996cabf3f336ecb33e74f790317bf21"} err="failed to get container status \"5c3165867b1f8900214e20a9394116a92996cabf3f336ecb33e74f790317bf21\": rpc error: code = NotFound desc = could not find container \"5c3165867b1f8900214e20a9394116a92996cabf3f336ecb33e74f790317bf21\": container with ID starting with 5c3165867b1f8900214e20a9394116a92996cabf3f336ecb33e74f790317bf21 not found: ID does not exist"
Sep 11 12:32:32.717327 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:32.717306    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn"]
Sep 11 12:32:32.723350 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:32.723327    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-82ead5-858f4979b9-f2fsn"]
Sep 11 12:32:32.932132 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:32.932097    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98ff6929-bdfa-4c6f-8ff0-819cc1376d45" path="/var/lib/kubelet/pods/98ff6929-bdfa-4c6f-8ff0-819cc1376d45/volumes"
Sep 11 12:32:35.657626 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:35.657592    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn"
Sep 11 12:32:35.657989 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:35.657769    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf"
Sep 11 12:32:35.706717 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:35.706683    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r" podUID="0a7d7bfe-b2cd-429f-a029-ca02ac169f76" containerName="main" containerID="cri-o://33bebf1bf0df0b2ce6e8f595a225c67350fcb5e13c4604de78602b41acb89cc3" gracePeriod=45
Sep 11 12:32:35.954061 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:35.954032    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:32:36.007475 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.007444    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-dshm\") pod \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\" (UID: \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\") "
Sep 11 12:32:36.007475 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.007481    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-tls-certs\") pod \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\" (UID: \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\") "
Sep 11 12:32:36.007655 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.007514    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-home\") pod \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\" (UID: \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\") "
Sep 11 12:32:36.007655 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.007634    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpfgp\" (UniqueName: \"kubernetes.io/projected/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-kube-api-access-qpfgp\") pod \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\" (UID: \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\") "
Sep 11 12:32:36.007779 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.007732    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-home" (OuterVolumeSpecName: "home") pod "0a7d7bfe-b2cd-429f-a029-ca02ac169f76" (UID: "0a7d7bfe-b2cd-429f-a029-ca02ac169f76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:32:36.007779 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.007757    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-tmp-dir\") pod \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\" (UID: \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\") "
Sep 11 12:32:36.007860 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.007800    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-model-cache\") pod \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\" (UID: \"0a7d7bfe-b2cd-429f-a029-ca02ac169f76\") "
Sep 11 12:32:36.008048 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.008018    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0a7d7bfe-b2cd-429f-a029-ca02ac169f76" (UID: "0a7d7bfe-b2cd-429f-a029-ca02ac169f76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:32:36.008182 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.008098    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-model-cache" (OuterVolumeSpecName: "model-cache") pod "0a7d7bfe-b2cd-429f-a029-ca02ac169f76" (UID: "0a7d7bfe-b2cd-429f-a029-ca02ac169f76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:32:36.008294 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.008251    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-tmp-dir\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:32:36.008294 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.008273    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-model-cache\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:32:36.008294 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.008286    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-home\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:32:36.009846 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.009820    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-kube-api-access-qpfgp" (OuterVolumeSpecName: "kube-api-access-qpfgp") pod "0a7d7bfe-b2cd-429f-a029-ca02ac169f76" (UID: "0a7d7bfe-b2cd-429f-a029-ca02ac169f76"). InnerVolumeSpecName "kube-api-access-qpfgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:32:36.009943 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.009839    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a7d7bfe-b2cd-429f-a029-ca02ac169f76" (UID: "0a7d7bfe-b2cd-429f-a029-ca02ac169f76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:32:36.009943 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.009887    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-dshm" (OuterVolumeSpecName: "dshm") pod "0a7d7bfe-b2cd-429f-a029-ca02ac169f76" (UID: "0a7d7bfe-b2cd-429f-a029-ca02ac169f76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:32:36.109024 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.108996    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-dshm\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:32:36.109024 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.109020    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-tls-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:32:36.109230 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.109032    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpfgp\" (UniqueName: \"kubernetes.io/projected/0a7d7bfe-b2cd-429f-a029-ca02ac169f76-kube-api-access-qpfgp\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:32:36.712401 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.712371    2561 generic.go:356] "Generic (PLEG): container finished" podID="0a7d7bfe-b2cd-429f-a029-ca02ac169f76" containerID="33bebf1bf0df0b2ce6e8f595a225c67350fcb5e13c4604de78602b41acb89cc3" exitCode=0
Sep 11 12:32:36.712840 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.712411    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"
Sep 11 12:32:36.712840 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.712420    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r" event={"ID":"0a7d7bfe-b2cd-429f-a029-ca02ac169f76","Type":"ContainerDied","Data":"33bebf1bf0df0b2ce6e8f595a225c67350fcb5e13c4604de78602b41acb89cc3"}
Sep 11 12:32:36.712840 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.712449    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r" event={"ID":"0a7d7bfe-b2cd-429f-a029-ca02ac169f76","Type":"ContainerDied","Data":"93fdae37db2c3408b6815d5cd08c7cadb2e76e863d2dd059790d5a755d175c11"}
Sep 11 12:32:36.712840 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.712466    2561 scope.go:117] "RemoveContainer" containerID="33bebf1bf0df0b2ce6e8f595a225c67350fcb5e13c4604de78602b41acb89cc3"
Sep 11 12:32:36.721958 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.721940    2561 scope.go:117] "RemoveContainer" containerID="33bebf1bf0df0b2ce6e8f595a225c67350fcb5e13c4604de78602b41acb89cc3"
Sep 11 12:32:36.722228 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:32:36.722208    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33bebf1bf0df0b2ce6e8f595a225c67350fcb5e13c4604de78602b41acb89cc3\": container with ID starting with 33bebf1bf0df0b2ce6e8f595a225c67350fcb5e13c4604de78602b41acb89cc3 not found: ID does not exist" containerID="33bebf1bf0df0b2ce6e8f595a225c67350fcb5e13c4604de78602b41acb89cc3"
Sep 11 12:32:36.722303 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.722236    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33bebf1bf0df0b2ce6e8f595a225c67350fcb5e13c4604de78602b41acb89cc3"} err="failed to get container status \"33bebf1bf0df0b2ce6e8f595a225c67350fcb5e13c4604de78602b41acb89cc3\": rpc error: code = NotFound desc = could not find container \"33bebf1bf0df0b2ce6e8f595a225c67350fcb5e13c4604de78602b41acb89cc3\": container with ID starting with 33bebf1bf0df0b2ce6e8f595a225c67350fcb5e13c4604de78602b41acb89cc3 not found: ID does not exist"
Sep 11 12:32:36.736722 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.736696    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"]
Sep 11 12:32:36.740916 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.740895    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-82ead5/explicit-model-82ead5-kserve-7c645d5b7f-6k99r"]
Sep 11 12:32:36.932665 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:36.932635    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a7d7bfe-b2cd-429f-a029-ca02ac169f76" path="/var/lib/kubelet/pods/0a7d7bfe-b2cd-429f-a029-ca02ac169f76/volumes"
Sep 11 12:32:43.377429 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.377394    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"]
Sep 11 12:32:43.377793 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.377759    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a7d7bfe-b2cd-429f-a029-ca02ac169f76" containerName="main"
Sep 11 12:32:43.377793 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.377769    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a7d7bfe-b2cd-429f-a029-ca02ac169f76" containerName="main"
Sep 11 12:32:43.377793 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.377781    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98ff6929-bdfa-4c6f-8ff0-819cc1376d45" containerName="payload-processing"
Sep 11 12:32:43.377793 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.377787    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="98ff6929-bdfa-4c6f-8ff0-819cc1376d45" containerName="payload-processing"
Sep 11 12:32:43.377928 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.377853    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a7d7bfe-b2cd-429f-a029-ca02ac169f76" containerName="main"
Sep 11 12:32:43.377928 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.377864    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="98ff6929-bdfa-4c6f-8ff0-819cc1376d45" containerName="payload-processing"
Sep 11 12:32:43.387566 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.387536    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:32:43.387970 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.387950    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"]
Sep 11 12:32:43.389957 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.389933    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cb20c8\"/\"e2e-conflict-a-97e93ffc-kserve-self-signed-certs\""
Sep 11 12:32:43.476093 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.476061    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/630eedab-4510-426b-9735-f9b976b5964c-dshm\") pod \"e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr\" (UID: \"630eedab-4510-426b-9735-f9b976b5964c\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:32:43.476264 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.476135    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630eedab-4510-426b-9735-f9b976b5964c-tmp-dir\") pod \"e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr\" (UID: \"630eedab-4510-426b-9735-f9b976b5964c\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:32:43.476264 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.476162    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/630eedab-4510-426b-9735-f9b976b5964c-model-cache\") pod \"e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr\" (UID: \"630eedab-4510-426b-9735-f9b976b5964c\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:32:43.476361 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.476265    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630eedab-4510-426b-9735-f9b976b5964c-tls-certs\") pod \"e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr\" (UID: \"630eedab-4510-426b-9735-f9b976b5964c\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:32:43.476361 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.476316    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5gn2\" (UniqueName: \"kubernetes.io/projected/630eedab-4510-426b-9735-f9b976b5964c-kube-api-access-v5gn2\") pod \"e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr\" (UID: \"630eedab-4510-426b-9735-f9b976b5964c\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:32:43.476448 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.476368    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/630eedab-4510-426b-9735-f9b976b5964c-home\") pod \"e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr\" (UID: \"630eedab-4510-426b-9735-f9b976b5964c\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:32:43.577718 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.577684    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/630eedab-4510-426b-9735-f9b976b5964c-model-cache\") pod \"e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr\" (UID: \"630eedab-4510-426b-9735-f9b976b5964c\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:32:43.577906 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.577756    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630eedab-4510-426b-9735-f9b976b5964c-tls-certs\") pod \"e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr\" (UID: \"630eedab-4510-426b-9735-f9b976b5964c\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:32:43.577906 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.577781    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5gn2\" (UniqueName: \"kubernetes.io/projected/630eedab-4510-426b-9735-f9b976b5964c-kube-api-access-v5gn2\") pod \"e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr\" (UID: \"630eedab-4510-426b-9735-f9b976b5964c\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:32:43.577906 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.577823    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/630eedab-4510-426b-9735-f9b976b5964c-home\") pod \"e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr\" (UID: \"630eedab-4510-426b-9735-f9b976b5964c\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:32:43.577906 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.577856    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/630eedab-4510-426b-9735-f9b976b5964c-dshm\") pod \"e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr\" (UID: \"630eedab-4510-426b-9735-f9b976b5964c\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:32:43.578119 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.577932    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630eedab-4510-426b-9735-f9b976b5964c-tmp-dir\") pod \"e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr\" (UID: \"630eedab-4510-426b-9735-f9b976b5964c\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:32:43.578119 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.578090    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/630eedab-4510-426b-9735-f9b976b5964c-model-cache\") pod \"e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr\" (UID: \"630eedab-4510-426b-9735-f9b976b5964c\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:32:43.578272 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.578182    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/630eedab-4510-426b-9735-f9b976b5964c-home\") pod \"e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr\" (UID: \"630eedab-4510-426b-9735-f9b976b5964c\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:32:43.578332 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.578299    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630eedab-4510-426b-9735-f9b976b5964c-tmp-dir\") pod \"e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr\" (UID: \"630eedab-4510-426b-9735-f9b976b5964c\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:32:43.580139 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.580108    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/630eedab-4510-426b-9735-f9b976b5964c-dshm\") pod \"e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr\" (UID: \"630eedab-4510-426b-9735-f9b976b5964c\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:32:43.580451 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.580429    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630eedab-4510-426b-9735-f9b976b5964c-tls-certs\") pod \"e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr\" (UID: \"630eedab-4510-426b-9735-f9b976b5964c\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:32:43.585458 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.585437    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5gn2\" (UniqueName: \"kubernetes.io/projected/630eedab-4510-426b-9735-f9b976b5964c-kube-api-access-v5gn2\") pod \"e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr\" (UID: \"630eedab-4510-426b-9735-f9b976b5964c\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:32:43.699665 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.699574    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:32:43.747657 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.747608    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:32:43.980789 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:43.980760    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"]
Sep 11 12:32:43.983074 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:32:43.983041    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod630eedab_4510_426b_9735_f9b976b5964c.slice/crio-6464a56048606c22ff57c5871cac9c4cb709b0f60b4fe4269296ba763ea03024 WatchSource:0}: Error finding container 6464a56048606c22ff57c5871cac9c4cb709b0f60b4fe4269296ba763ea03024: Status 404 returned error can't find the container with id 6464a56048606c22ff57c5871cac9c4cb709b0f60b4fe4269296ba763ea03024
Sep 11 12:32:44.754465 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:44.754419    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr" event={"ID":"630eedab-4510-426b-9735-f9b976b5964c","Type":"ContainerStarted","Data":"e50c72467019a3feeab10c0ec01f9a668e5c139b8de062d7ed71b0329b88a24c"}
Sep 11 12:32:44.754465 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:44.754461    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr" event={"ID":"630eedab-4510-426b-9735-f9b976b5964c","Type":"ContainerStarted","Data":"6464a56048606c22ff57c5871cac9c4cb709b0f60b4fe4269296ba763ea03024"}
Sep 11 12:32:44.771839 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:44.771791    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr" podStartSLOduration=1.771776894 podStartE2EDuration="1.771776894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:32:44.770675965 +0000 UTC m=+1096.333455732" watchObservedRunningTime="2026-09-11 12:32:44.771776894 +0000 UTC m=+1096.334556714"
Sep 11 12:32:53.699785 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:53.699753    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:32:53.700229 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:53.699802    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:32:53.712469 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:53.712442    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:32:53.809569 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:53.809540    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:32:56.883805 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:56.883772    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"]
Sep 11 12:32:56.887005 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:56.886989    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:32:56.889489 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:56.889459    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1a8ee3\"/\"distinct-w0-1a8ee3-kserve-self-signed-certs\""
Sep 11 12:32:56.895569 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:56.895549    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"]
Sep 11 12:32:57.008349 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:57.008313    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb3646b9-acb4-43d6-8f30-768306fdb4b3-tls-certs\") pod \"distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58\" (UID: \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:32:57.008349 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:57.008349    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g7hv\" (UniqueName: \"kubernetes.io/projected/fb3646b9-acb4-43d6-8f30-768306fdb4b3-kube-api-access-7g7hv\") pod \"distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58\" (UID: \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:32:57.008552 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:57.008372    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb3646b9-acb4-43d6-8f30-768306fdb4b3-model-cache\") pod \"distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58\" (UID: \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:32:57.008552 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:57.008465    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb3646b9-acb4-43d6-8f30-768306fdb4b3-dshm\") pod \"distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58\" (UID: \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:32:57.008552 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:57.008508    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb3646b9-acb4-43d6-8f30-768306fdb4b3-home\") pod \"distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58\" (UID: \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:32:57.008662 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:57.008584    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb3646b9-acb4-43d6-8f30-768306fdb4b3-tmp-dir\") pod \"distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58\" (UID: \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:32:57.109806 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:57.109773    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb3646b9-acb4-43d6-8f30-768306fdb4b3-home\") pod \"distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58\" (UID: \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:32:57.109936 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:57.109825    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb3646b9-acb4-43d6-8f30-768306fdb4b3-tmp-dir\") pod \"distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58\" (UID: \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:32:57.109936 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:57.109895    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb3646b9-acb4-43d6-8f30-768306fdb4b3-tls-certs\") pod \"distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58\" (UID: \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:32:57.109936 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:57.109922    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g7hv\" (UniqueName: \"kubernetes.io/projected/fb3646b9-acb4-43d6-8f30-768306fdb4b3-kube-api-access-7g7hv\") pod \"distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58\" (UID: \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:32:57.110097 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:57.109955    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb3646b9-acb4-43d6-8f30-768306fdb4b3-model-cache\") pod \"distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58\" (UID: \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:32:57.110097 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:57.110035    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb3646b9-acb4-43d6-8f30-768306fdb4b3-dshm\") pod \"distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58\" (UID: \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:32:57.110240 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:57.110218    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb3646b9-acb4-43d6-8f30-768306fdb4b3-home\") pod \"distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58\" (UID: \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:32:57.110310 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:57.110235    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb3646b9-acb4-43d6-8f30-768306fdb4b3-tmp-dir\") pod \"distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58\" (UID: \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:32:57.110465 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:57.110438    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb3646b9-acb4-43d6-8f30-768306fdb4b3-model-cache\") pod \"distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58\" (UID: \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:32:57.112126 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:57.112102    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb3646b9-acb4-43d6-8f30-768306fdb4b3-dshm\") pod \"distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58\" (UID: \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:32:57.112272 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:57.112255    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb3646b9-acb4-43d6-8f30-768306fdb4b3-tls-certs\") pod \"distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58\" (UID: \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:32:57.118470 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:57.118449    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g7hv\" (UniqueName: \"kubernetes.io/projected/fb3646b9-acb4-43d6-8f30-768306fdb4b3-kube-api-access-7g7hv\") pod \"distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58\" (UID: \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:32:57.197395 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:57.197338    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:32:57.470654 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:57.470627    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"]
Sep 11 12:32:57.472173 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:32:57.472149    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb3646b9_acb4_43d6_8f30_768306fdb4b3.slice/crio-e00a223fcb70c91baa99172a23c12a661eacd568d4379d83a3ece41390e0b784 WatchSource:0}: Error finding container e00a223fcb70c91baa99172a23c12a661eacd568d4379d83a3ece41390e0b784: Status 404 returned error can't find the container with id e00a223fcb70c91baa99172a23c12a661eacd568d4379d83a3ece41390e0b784
Sep 11 12:32:57.819581 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:57.819545    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58" event={"ID":"fb3646b9-acb4-43d6-8f30-768306fdb4b3","Type":"ContainerStarted","Data":"42983d8339d2c5a8b63244882b1cfaf484c6aec5eff91849a9714f19a38edd25"}
Sep 11 12:32:57.819581 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:57.819585    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58" event={"ID":"fb3646b9-acb4-43d6-8f30-768306fdb4b3","Type":"ContainerStarted","Data":"e00a223fcb70c91baa99172a23c12a661eacd568d4379d83a3ece41390e0b784"}
Sep 11 12:32:57.841910 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:32:57.841868    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58" podStartSLOduration=1.841853693 podStartE2EDuration="1.841853693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:32:57.839392218 +0000 UTC m=+1109.402171981" watchObservedRunningTime="2026-09-11 12:32:57.841853693 +0000 UTC m=+1109.404633457"
Sep 11 12:33:04.800252 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:04.800217    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf"]
Sep 11 12:33:04.800747 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:04.800433    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf" podUID="b8b6d48e-adfc-461d-92a7-42b0857f4b10" containerName="payload-processing" containerID="cri-o://47ad0d52fd31ec1026a7bb7496cfc3976f9e0292cef51d0249a4c8f746cf7ad0" gracePeriod=30
Sep 11 12:33:04.810279 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:04.809683    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn"]
Sep 11 12:33:04.810279 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:04.809901    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn" podUID="14a89d6e-f0a1-4eb5-a801-ad2e64f282a0" containerName="payload-pre-processing" containerID="cri-o://3943e3d9fb90a3df448ddc06b141dcd38a3daa96883c21d88cb26afd0e9bff34" gracePeriod=30
Sep 11 12:33:05.080859 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.080824    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf"
Sep 11 12:33:05.083782 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.083762    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn"
Sep 11 12:33:05.085259 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.085236    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xff78\" (UniqueName: \"kubernetes.io/projected/b8b6d48e-adfc-461d-92a7-42b0857f4b10-kube-api-access-xff78\") pod \"b8b6d48e-adfc-461d-92a7-42b0857f4b10\" (UID: \"b8b6d48e-adfc-461d-92a7-42b0857f4b10\") "
Sep 11 12:33:05.085398 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.085371    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b6d48e-adfc-461d-92a7-42b0857f4b10-plugins-config-volume\") pod \"b8b6d48e-adfc-461d-92a7-42b0857f4b10\" (UID: \"b8b6d48e-adfc-461d-92a7-42b0857f4b10\") "
Sep 11 12:33:05.085743 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.085721    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b6d48e-adfc-461d-92a7-42b0857f4b10-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b8b6d48e-adfc-461d-92a7-42b0857f4b10" (UID: "b8b6d48e-adfc-461d-92a7-42b0857f4b10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:33:05.087252 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.087224    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b6d48e-adfc-461d-92a7-42b0857f4b10-kube-api-access-xff78" (OuterVolumeSpecName: "kube-api-access-xff78") pod "b8b6d48e-adfc-461d-92a7-42b0857f4b10" (UID: "b8b6d48e-adfc-461d-92a7-42b0857f4b10"). InnerVolumeSpecName "kube-api-access-xff78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:33:05.186723 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.186696    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5d6w9\" (UniqueName: \"kubernetes.io/projected/14a89d6e-f0a1-4eb5-a801-ad2e64f282a0-kube-api-access-5d6w9\") pod \"14a89d6e-f0a1-4eb5-a801-ad2e64f282a0\" (UID: \"14a89d6e-f0a1-4eb5-a801-ad2e64f282a0\") "
Sep 11 12:33:05.186908 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.186762    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a89d6e-f0a1-4eb5-a801-ad2e64f282a0-plugins-config-volume\") pod \"14a89d6e-f0a1-4eb5-a801-ad2e64f282a0\" (UID: \"14a89d6e-f0a1-4eb5-a801-ad2e64f282a0\") "
Sep 11 12:33:05.186977 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.186945    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b6d48e-adfc-461d-92a7-42b0857f4b10-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:33:05.186977 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.186956    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xff78\" (UniqueName: \"kubernetes.io/projected/b8b6d48e-adfc-461d-92a7-42b0857f4b10-kube-api-access-xff78\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:33:05.187158 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.187134    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a89d6e-f0a1-4eb5-a801-ad2e64f282a0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "14a89d6e-f0a1-4eb5-a801-ad2e64f282a0" (UID: "14a89d6e-f0a1-4eb5-a801-ad2e64f282a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:33:05.188786 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.188765    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a89d6e-f0a1-4eb5-a801-ad2e64f282a0-kube-api-access-5d6w9" (OuterVolumeSpecName: "kube-api-access-5d6w9") pod "14a89d6e-f0a1-4eb5-a801-ad2e64f282a0" (UID: "14a89d6e-f0a1-4eb5-a801-ad2e64f282a0"). InnerVolumeSpecName "kube-api-access-5d6w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:33:05.288313 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.288277    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a89d6e-f0a1-4eb5-a801-ad2e64f282a0-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:33:05.288313 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.288306    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d6w9\" (UniqueName: \"kubernetes.io/projected/14a89d6e-f0a1-4eb5-a801-ad2e64f282a0-kube-api-access-5d6w9\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:33:05.860973 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.860944    2561 generic.go:356] "Generic (PLEG): container finished" podID="b8b6d48e-adfc-461d-92a7-42b0857f4b10" containerID="47ad0d52fd31ec1026a7bb7496cfc3976f9e0292cef51d0249a4c8f746cf7ad0" exitCode=0
Sep 11 12:33:05.861424 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.860989    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf"
Sep 11 12:33:05.861424 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.861028    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf" event={"ID":"b8b6d48e-adfc-461d-92a7-42b0857f4b10","Type":"ContainerDied","Data":"47ad0d52fd31ec1026a7bb7496cfc3976f9e0292cef51d0249a4c8f746cf7ad0"}
Sep 11 12:33:05.861424 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.861083    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf" event={"ID":"b8b6d48e-adfc-461d-92a7-42b0857f4b10","Type":"ContainerDied","Data":"a30c960d911be7819c3292ba764cf325c9f556bccf5dc70855a54f0db8ef3dc7"}
Sep 11 12:33:05.861424 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.861138    2561 scope.go:117] "RemoveContainer" containerID="47ad0d52fd31ec1026a7bb7496cfc3976f9e0292cef51d0249a4c8f746cf7ad0"
Sep 11 12:33:05.862474 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.862453    2561 generic.go:356] "Generic (PLEG): container finished" podID="14a89d6e-f0a1-4eb5-a801-ad2e64f282a0" containerID="3943e3d9fb90a3df448ddc06b141dcd38a3daa96883c21d88cb26afd0e9bff34" exitCode=0
Sep 11 12:33:05.862536 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.862499    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn"
Sep 11 12:33:05.862590 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.862555    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn" event={"ID":"14a89d6e-f0a1-4eb5-a801-ad2e64f282a0","Type":"ContainerDied","Data":"3943e3d9fb90a3df448ddc06b141dcd38a3daa96883c21d88cb26afd0e9bff34"}
Sep 11 12:33:05.862644 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.862600    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn" event={"ID":"14a89d6e-f0a1-4eb5-a801-ad2e64f282a0","Type":"ContainerDied","Data":"f2264b4178963fbdfbd31b92dc2bb11220bd01eed948167a4e5c28dbbc7648d7"}
Sep 11 12:33:05.870429 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.870409    2561 scope.go:117] "RemoveContainer" containerID="47ad0d52fd31ec1026a7bb7496cfc3976f9e0292cef51d0249a4c8f746cf7ad0"
Sep 11 12:33:05.870699 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:33:05.870679    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47ad0d52fd31ec1026a7bb7496cfc3976f9e0292cef51d0249a4c8f746cf7ad0\": container with ID starting with 47ad0d52fd31ec1026a7bb7496cfc3976f9e0292cef51d0249a4c8f746cf7ad0 not found: ID does not exist" containerID="47ad0d52fd31ec1026a7bb7496cfc3976f9e0292cef51d0249a4c8f746cf7ad0"
Sep 11 12:33:05.870777 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.870706    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47ad0d52fd31ec1026a7bb7496cfc3976f9e0292cef51d0249a4c8f746cf7ad0"} err="failed to get container status \"47ad0d52fd31ec1026a7bb7496cfc3976f9e0292cef51d0249a4c8f746cf7ad0\": rpc error: code = NotFound desc = could not find container \"47ad0d52fd31ec1026a7bb7496cfc3976f9e0292cef51d0249a4c8f746cf7ad0\": container with ID starting with 47ad0d52fd31ec1026a7bb7496cfc3976f9e0292cef51d0249a4c8f746cf7ad0 not found: ID does not exist"
Sep 11 12:33:05.870777 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.870724    2561 scope.go:117] "RemoveContainer" containerID="3943e3d9fb90a3df448ddc06b141dcd38a3daa96883c21d88cb26afd0e9bff34"
Sep 11 12:33:05.878692 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.878675    2561 scope.go:117] "RemoveContainer" containerID="3943e3d9fb90a3df448ddc06b141dcd38a3daa96883c21d88cb26afd0e9bff34"
Sep 11 12:33:05.878928 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:33:05.878911    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3943e3d9fb90a3df448ddc06b141dcd38a3daa96883c21d88cb26afd0e9bff34\": container with ID starting with 3943e3d9fb90a3df448ddc06b141dcd38a3daa96883c21d88cb26afd0e9bff34 not found: ID does not exist" containerID="3943e3d9fb90a3df448ddc06b141dcd38a3daa96883c21d88cb26afd0e9bff34"
Sep 11 12:33:05.878987 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.878935    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3943e3d9fb90a3df448ddc06b141dcd38a3daa96883c21d88cb26afd0e9bff34"} err="failed to get container status \"3943e3d9fb90a3df448ddc06b141dcd38a3daa96883c21d88cb26afd0e9bff34\": rpc error: code = NotFound desc = could not find container \"3943e3d9fb90a3df448ddc06b141dcd38a3daa96883c21d88cb26afd0e9bff34\": container with ID starting with 3943e3d9fb90a3df448ddc06b141dcd38a3daa96883c21d88cb26afd0e9bff34 not found: ID does not exist"
Sep 11 12:33:05.886653 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.886633    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn"]
Sep 11 12:33:05.896888 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.894881    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ad618118-597fc7bb7d-tgnwn"]
Sep 11 12:33:05.907547 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.907527    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf"]
Sep 11 12:33:05.913476 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:05.913453    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ad618118-6899f9c697-qcwtf"]
Sep 11 12:33:06.932261 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:06.932227    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14a89d6e-f0a1-4eb5-a801-ad2e64f282a0" path="/var/lib/kubelet/pods/14a89d6e-f0a1-4eb5-a801-ad2e64f282a0/volumes"
Sep 11 12:33:06.932775 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:06.932752    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8b6d48e-adfc-461d-92a7-42b0857f4b10" path="/var/lib/kubelet/pods/b8b6d48e-adfc-461d-92a7-42b0857f4b10/volumes"
Sep 11 12:33:07.198080 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:07.197983    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:33:07.198080 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:07.198037    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:33:07.210539 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:07.210513    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:33:07.885958 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:07.885927    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:33:12.746643 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:12.746607    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:33:25.590404 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:25.590370    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"]
Sep 11 12:33:30.184410 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:30.184374    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr"]
Sep 11 12:33:30.184788 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:30.184748    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14a89d6e-f0a1-4eb5-a801-ad2e64f282a0" containerName="payload-pre-processing"
Sep 11 12:33:30.184788 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:30.184759    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="14a89d6e-f0a1-4eb5-a801-ad2e64f282a0" containerName="payload-pre-processing"
Sep 11 12:33:30.184788 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:30.184783    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b6d48e-adfc-461d-92a7-42b0857f4b10" containerName="payload-processing"
Sep 11 12:33:30.184788 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:30.184789    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b6d48e-adfc-461d-92a7-42b0857f4b10" containerName="payload-processing"
Sep 11 12:33:30.184918 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:30.184857    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="14a89d6e-f0a1-4eb5-a801-ad2e64f282a0" containerName="payload-pre-processing"
Sep 11 12:33:30.184918 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:30.184864    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b6d48e-adfc-461d-92a7-42b0857f4b10" containerName="payload-processing"
Sep 11 12:33:30.189395 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:30.189378    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr"
Sep 11 12:33:30.192045 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:30.192024    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-44f372-dockercfg-nggr9\""
Sep 11 12:33:30.192160 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:30.192082    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-44f372\""
Sep 11 12:33:30.198141 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:30.198115    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr"]
Sep 11 12:33:30.324238 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:30.324181    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdsj8\" (UniqueName: \"kubernetes.io/projected/8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778-kube-api-access-bdsj8\") pod \"payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr\" (UID: \"8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr"
Sep 11 12:33:30.324402 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:30.324314    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr\" (UID: \"8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr"
Sep 11 12:33:30.425329 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:30.425288    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr\" (UID: \"8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr"
Sep 11 12:33:30.425464 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:30.425381    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdsj8\" (UniqueName: \"kubernetes.io/projected/8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778-kube-api-access-bdsj8\") pod \"payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr\" (UID: \"8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr"
Sep 11 12:33:30.425984 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:30.425963    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr\" (UID: \"8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr"
Sep 11 12:33:30.436911 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:30.436849    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdsj8\" (UniqueName: \"kubernetes.io/projected/8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778-kube-api-access-bdsj8\") pod \"payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr\" (UID: \"8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr"
Sep 11 12:33:30.499983 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:30.499941    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr"
Sep 11 12:33:30.776895 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:30.776870    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr"]
Sep 11 12:33:30.778854 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:33:30.778827    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aad8178_8fe7_4b6d_9cb4_5ec1fc3ac778.slice/crio-3ea7b12d674bb3adb30822c43015c7f6a1481be191f9414ba0d59152d58ccadd WatchSource:0}: Error finding container 3ea7b12d674bb3adb30822c43015c7f6a1481be191f9414ba0d59152d58ccadd: Status 404 returned error can't find the container with id 3ea7b12d674bb3adb30822c43015c7f6a1481be191f9414ba0d59152d58ccadd
Sep 11 12:33:30.990562 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:30.990521    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr" event={"ID":"8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778","Type":"ContainerStarted","Data":"382ed57ad055da90383bb6e92da04891f64e7794699fd4a78c20d7146708bd3a"}
Sep 11 12:33:30.990743 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:30.990583    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr"
Sep 11 12:33:30.990743 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:30.990607    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr" event={"ID":"8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778","Type":"ContainerStarted","Data":"3ea7b12d674bb3adb30822c43015c7f6a1481be191f9414ba0d59152d58ccadd"}
Sep 11 12:33:31.007338 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:31.007291    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr" podStartSLOduration=1.007276406 podStartE2EDuration="1.007276406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:33:31.006763374 +0000 UTC m=+1142.569543139" watchObservedRunningTime="2026-09-11 12:33:31.007276406 +0000 UTC m=+1142.570056169"
Sep 11 12:33:40.611338 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:40.611281    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb" podUID="402a229f-9035-408d-a896-f54fd8871ef1" containerName="main" containerID="cri-o://f471b551b85da05471a1bb2aaafb6d7543fd5ceb8cca8d76b8aae261d2bc366b" gracePeriod=45
Sep 11 12:33:40.863220 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:40.863152    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:33:41.023482 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.023449    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/402a229f-9035-408d-a896-f54fd8871ef1-tls-certs\") pod \"402a229f-9035-408d-a896-f54fd8871ef1\" (UID: \"402a229f-9035-408d-a896-f54fd8871ef1\") "
Sep 11 12:33:41.023482 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.023489    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7t59b\" (UniqueName: \"kubernetes.io/projected/402a229f-9035-408d-a896-f54fd8871ef1-kube-api-access-7t59b\") pod \"402a229f-9035-408d-a896-f54fd8871ef1\" (UID: \"402a229f-9035-408d-a896-f54fd8871ef1\") "
Sep 11 12:33:41.023687 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.023522    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/402a229f-9035-408d-a896-f54fd8871ef1-dshm\") pod \"402a229f-9035-408d-a896-f54fd8871ef1\" (UID: \"402a229f-9035-408d-a896-f54fd8871ef1\") "
Sep 11 12:33:41.023687 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.023654    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/402a229f-9035-408d-a896-f54fd8871ef1-home\") pod \"402a229f-9035-408d-a896-f54fd8871ef1\" (UID: \"402a229f-9035-408d-a896-f54fd8871ef1\") "
Sep 11 12:33:41.023761 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.023715    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/402a229f-9035-408d-a896-f54fd8871ef1-model-cache\") pod \"402a229f-9035-408d-a896-f54fd8871ef1\" (UID: \"402a229f-9035-408d-a896-f54fd8871ef1\") "
Sep 11 12:33:41.023811 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.023800    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/402a229f-9035-408d-a896-f54fd8871ef1-tmp-dir\") pod \"402a229f-9035-408d-a896-f54fd8871ef1\" (UID: \"402a229f-9035-408d-a896-f54fd8871ef1\") "
Sep 11 12:33:41.023973 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.023943    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402a229f-9035-408d-a896-f54fd8871ef1-home" (OuterVolumeSpecName: "home") pod "402a229f-9035-408d-a896-f54fd8871ef1" (UID: "402a229f-9035-408d-a896-f54fd8871ef1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:33:41.024129 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.023980    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402a229f-9035-408d-a896-f54fd8871ef1-model-cache" (OuterVolumeSpecName: "model-cache") pod "402a229f-9035-408d-a896-f54fd8871ef1" (UID: "402a229f-9035-408d-a896-f54fd8871ef1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:33:41.024129 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.024089    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402a229f-9035-408d-a896-f54fd8871ef1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "402a229f-9035-408d-a896-f54fd8871ef1" (UID: "402a229f-9035-408d-a896-f54fd8871ef1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:33:41.024288 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.024268    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/402a229f-9035-408d-a896-f54fd8871ef1-tmp-dir\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:33:41.024346 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.024289    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/402a229f-9035-408d-a896-f54fd8871ef1-home\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:33:41.024346 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.024303    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/402a229f-9035-408d-a896-f54fd8871ef1-model-cache\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:33:41.025869 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.025845    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402a229f-9035-408d-a896-f54fd8871ef1-dshm" (OuterVolumeSpecName: "dshm") pod "402a229f-9035-408d-a896-f54fd8871ef1" (UID: "402a229f-9035-408d-a896-f54fd8871ef1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:33:41.026175 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.026155    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/402a229f-9035-408d-a896-f54fd8871ef1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "402a229f-9035-408d-a896-f54fd8871ef1" (UID: "402a229f-9035-408d-a896-f54fd8871ef1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:33:41.026260 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.026168    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402a229f-9035-408d-a896-f54fd8871ef1-kube-api-access-7t59b" (OuterVolumeSpecName: "kube-api-access-7t59b") pod "402a229f-9035-408d-a896-f54fd8871ef1" (UID: "402a229f-9035-408d-a896-f54fd8871ef1"). InnerVolumeSpecName "kube-api-access-7t59b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:33:41.040492 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.040468    2561 generic.go:356] "Generic (PLEG): container finished" podID="402a229f-9035-408d-a896-f54fd8871ef1" containerID="f471b551b85da05471a1bb2aaafb6d7543fd5ceb8cca8d76b8aae261d2bc366b" exitCode=0
Sep 11 12:33:41.040597 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.040506    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"
Sep 11 12:33:41.040597 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.040549    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb" event={"ID":"402a229f-9035-408d-a896-f54fd8871ef1","Type":"ContainerDied","Data":"f471b551b85da05471a1bb2aaafb6d7543fd5ceb8cca8d76b8aae261d2bc366b"}
Sep 11 12:33:41.040597 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.040586    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb" event={"ID":"402a229f-9035-408d-a896-f54fd8871ef1","Type":"ContainerDied","Data":"7898f17800ef956d80b606568ef74c10d8820f234a1ffbdb836c459465856547"}
Sep 11 12:33:41.040734 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.040605    2561 scope.go:117] "RemoveContainer" containerID="f471b551b85da05471a1bb2aaafb6d7543fd5ceb8cca8d76b8aae261d2bc366b"
Sep 11 12:33:41.049040 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.049023    2561 scope.go:117] "RemoveContainer" containerID="f471b551b85da05471a1bb2aaafb6d7543fd5ceb8cca8d76b8aae261d2bc366b"
Sep 11 12:33:41.049274 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:33:41.049253    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f471b551b85da05471a1bb2aaafb6d7543fd5ceb8cca8d76b8aae261d2bc366b\": container with ID starting with f471b551b85da05471a1bb2aaafb6d7543fd5ceb8cca8d76b8aae261d2bc366b not found: ID does not exist" containerID="f471b551b85da05471a1bb2aaafb6d7543fd5ceb8cca8d76b8aae261d2bc366b"
Sep 11 12:33:41.049373 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.049281    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f471b551b85da05471a1bb2aaafb6d7543fd5ceb8cca8d76b8aae261d2bc366b"} err="failed to get container status \"f471b551b85da05471a1bb2aaafb6d7543fd5ceb8cca8d76b8aae261d2bc366b\": rpc error: code = NotFound desc = could not find container \"f471b551b85da05471a1bb2aaafb6d7543fd5ceb8cca8d76b8aae261d2bc366b\": container with ID starting with f471b551b85da05471a1bb2aaafb6d7543fd5ceb8cca8d76b8aae261d2bc366b not found: ID does not exist"
Sep 11 12:33:41.062147 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.062128    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"]
Sep 11 12:33:41.066978 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.066957    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-38a724/test-model-38a724-kserve-688b76459f-tzxwb"]
Sep 11 12:33:41.125650 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.125590    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/402a229f-9035-408d-a896-f54fd8871ef1-tls-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:33:41.125650 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.125614    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t59b\" (UniqueName: \"kubernetes.io/projected/402a229f-9035-408d-a896-f54fd8871ef1-kube-api-access-7t59b\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:33:41.125650 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.125623    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/402a229f-9035-408d-a896-f54fd8871ef1-dshm\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:33:41.997684 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:41.997653    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr"
Sep 11 12:33:42.932624 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:42.932589    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="402a229f-9035-408d-a896-f54fd8871ef1" path="/var/lib/kubelet/pods/402a229f-9035-408d-a896-f54fd8871ef1/volumes"
Sep 11 12:33:48.936465 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:48.936434    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:33:51.624159 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:51.624076    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb"]
Sep 11 12:33:51.624631 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:51.624306    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb" podUID="0a2242f0-9987-4537-82ef-27d33f94c69e" containerName="payload-processing" containerID="cri-o://08356ed73342bfc424ae8ca536fdb34295c9916e76e272add07ccf999587b00a" gracePeriod=30
Sep 11 12:33:51.876284 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:51.876221    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb"
Sep 11 12:33:51.920713 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:51.920685    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2242f0-9987-4537-82ef-27d33f94c69e-plugins-config-volume\") pod \"0a2242f0-9987-4537-82ef-27d33f94c69e\" (UID: \"0a2242f0-9987-4537-82ef-27d33f94c69e\") "
Sep 11 12:33:51.920877 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:51.920798    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfz6h\" (UniqueName: \"kubernetes.io/projected/0a2242f0-9987-4537-82ef-27d33f94c69e-kube-api-access-bfz6h\") pod \"0a2242f0-9987-4537-82ef-27d33f94c69e\" (UID: \"0a2242f0-9987-4537-82ef-27d33f94c69e\") "
Sep 11 12:33:51.921089 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:51.921063    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a2242f0-9987-4537-82ef-27d33f94c69e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0a2242f0-9987-4537-82ef-27d33f94c69e" (UID: "0a2242f0-9987-4537-82ef-27d33f94c69e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:33:51.922826 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:51.922799    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2242f0-9987-4537-82ef-27d33f94c69e-kube-api-access-bfz6h" (OuterVolumeSpecName: "kube-api-access-bfz6h") pod "0a2242f0-9987-4537-82ef-27d33f94c69e" (UID: "0a2242f0-9987-4537-82ef-27d33f94c69e"). InnerVolumeSpecName "kube-api-access-bfz6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:33:52.022598 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:52.022565    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfz6h\" (UniqueName: \"kubernetes.io/projected/0a2242f0-9987-4537-82ef-27d33f94c69e-kube-api-access-bfz6h\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:33:52.022598 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:52.022588    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2242f0-9987-4537-82ef-27d33f94c69e-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:33:52.097661 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:52.097630    2561 generic.go:356] "Generic (PLEG): container finished" podID="0a2242f0-9987-4537-82ef-27d33f94c69e" containerID="08356ed73342bfc424ae8ca536fdb34295c9916e76e272add07ccf999587b00a" exitCode=0
Sep 11 12:33:52.097787 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:52.097675    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb"
Sep 11 12:33:52.097787 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:52.097717    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb" event={"ID":"0a2242f0-9987-4537-82ef-27d33f94c69e","Type":"ContainerDied","Data":"08356ed73342bfc424ae8ca536fdb34295c9916e76e272add07ccf999587b00a"}
Sep 11 12:33:52.097787 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:52.097761    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb" event={"ID":"0a2242f0-9987-4537-82ef-27d33f94c69e","Type":"ContainerDied","Data":"96b89ee7f0b4860774145d03bc093ad8ef1c4732c3e1ce8a4371239e9136382b"}
Sep 11 12:33:52.097787 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:52.097782    2561 scope.go:117] "RemoveContainer" containerID="08356ed73342bfc424ae8ca536fdb34295c9916e76e272add07ccf999587b00a"
Sep 11 12:33:52.111551 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:52.111529    2561 scope.go:117] "RemoveContainer" containerID="08356ed73342bfc424ae8ca536fdb34295c9916e76e272add07ccf999587b00a"
Sep 11 12:33:52.111843 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:33:52.111826    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08356ed73342bfc424ae8ca536fdb34295c9916e76e272add07ccf999587b00a\": container with ID starting with 08356ed73342bfc424ae8ca536fdb34295c9916e76e272add07ccf999587b00a not found: ID does not exist" containerID="08356ed73342bfc424ae8ca536fdb34295c9916e76e272add07ccf999587b00a"
Sep 11 12:33:52.111907 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:52.111853    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08356ed73342bfc424ae8ca536fdb34295c9916e76e272add07ccf999587b00a"} err="failed to get container status \"08356ed73342bfc424ae8ca536fdb34295c9916e76e272add07ccf999587b00a\": rpc error: code = NotFound desc = could not find container \"08356ed73342bfc424ae8ca536fdb34295c9916e76e272add07ccf999587b00a\": container with ID starting with 08356ed73342bfc424ae8ca536fdb34295c9916e76e272add07ccf999587b00a not found: ID does not exist"
Sep 11 12:33:52.123780 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:52.123755    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb"]
Sep 11 12:33:52.129914 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:52.129856    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-38a724-d5954b995-x8tjb"]
Sep 11 12:33:52.932310 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:52.932260    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a2242f0-9987-4537-82ef-27d33f94c69e" path="/var/lib/kubelet/pods/0a2242f0-9987-4537-82ef-27d33f94c69e/volumes"
Sep 11 12:33:58.584635 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.584597    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt"]
Sep 11 12:33:58.585004 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.584946    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a2242f0-9987-4537-82ef-27d33f94c69e" containerName="payload-processing"
Sep 11 12:33:58.585004 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.584956    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a2242f0-9987-4537-82ef-27d33f94c69e" containerName="payload-processing"
Sep 11 12:33:58.585004 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.584976    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="402a229f-9035-408d-a896-f54fd8871ef1" containerName="main"
Sep 11 12:33:58.585004 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.584981    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="402a229f-9035-408d-a896-f54fd8871ef1" containerName="main"
Sep 11 12:33:58.585149 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.585050    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a2242f0-9987-4537-82ef-27d33f94c69e" containerName="payload-processing"
Sep 11 12:33:58.585149 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.585058    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="402a229f-9035-408d-a896-f54fd8871ef1" containerName="main"
Sep 11 12:33:58.587755 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.587739    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt"
Sep 11 12:33:58.590257 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.590235    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-a50cf9\""
Sep 11 12:33:58.598638 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.598615    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt"]
Sep 11 12:33:58.614890 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.614868    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq"]
Sep 11 12:33:58.618076 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.618060    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq"
Sep 11 12:33:58.620467 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.620447    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-a50cf9-dockercfg-w9rsm\""
Sep 11 12:33:58.630266 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.630244    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq"]
Sep 11 12:33:58.683352 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.683328    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwzqr\" (UniqueName: \"kubernetes.io/projected/6d179607-e322-4187-86d5-e3af6a0b1b60-kube-api-access-mwzqr\") pod \"payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt\" (UID: \"6d179607-e322-4187-86d5-e3af6a0b1b60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt"
Sep 11 12:33:58.683489 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.683368    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4dcq\" (UniqueName: \"kubernetes.io/projected/630ceb15-2cf6-4268-9e8e-27d6c2530fc7-kube-api-access-r4dcq\") pod \"payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq\" (UID: \"630ceb15-2cf6-4268-9e8e-27d6c2530fc7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq"
Sep 11 12:33:58.683489 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.683391    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d179607-e322-4187-86d5-e3af6a0b1b60-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt\" (UID: \"6d179607-e322-4187-86d5-e3af6a0b1b60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt"
Sep 11 12:33:58.683582 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.683534    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/630ceb15-2cf6-4268-9e8e-27d6c2530fc7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq\" (UID: \"630ceb15-2cf6-4268-9e8e-27d6c2530fc7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq"
Sep 11 12:33:58.784324 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.784293    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4dcq\" (UniqueName: \"kubernetes.io/projected/630ceb15-2cf6-4268-9e8e-27d6c2530fc7-kube-api-access-r4dcq\") pod \"payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq\" (UID: \"630ceb15-2cf6-4268-9e8e-27d6c2530fc7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq"
Sep 11 12:33:58.784454 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.784332    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d179607-e322-4187-86d5-e3af6a0b1b60-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt\" (UID: \"6d179607-e322-4187-86d5-e3af6a0b1b60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt"
Sep 11 12:33:58.784454 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.784406    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/630ceb15-2cf6-4268-9e8e-27d6c2530fc7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq\" (UID: \"630ceb15-2cf6-4268-9e8e-27d6c2530fc7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq"
Sep 11 12:33:58.784454 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.784452    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwzqr\" (UniqueName: \"kubernetes.io/projected/6d179607-e322-4187-86d5-e3af6a0b1b60-kube-api-access-mwzqr\") pod \"payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt\" (UID: \"6d179607-e322-4187-86d5-e3af6a0b1b60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt"
Sep 11 12:33:58.785338 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.785315    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/630ceb15-2cf6-4268-9e8e-27d6c2530fc7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq\" (UID: \"630ceb15-2cf6-4268-9e8e-27d6c2530fc7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq"
Sep 11 12:33:58.785398 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.785315    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d179607-e322-4187-86d5-e3af6a0b1b60-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt\" (UID: \"6d179607-e322-4187-86d5-e3af6a0b1b60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt"
Sep 11 12:33:58.798258 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.798230    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4dcq\" (UniqueName: \"kubernetes.io/projected/630ceb15-2cf6-4268-9e8e-27d6c2530fc7-kube-api-access-r4dcq\") pod \"payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq\" (UID: \"630ceb15-2cf6-4268-9e8e-27d6c2530fc7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq"
Sep 11 12:33:58.799379 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.799361    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwzqr\" (UniqueName: \"kubernetes.io/projected/6d179607-e322-4187-86d5-e3af6a0b1b60-kube-api-access-mwzqr\") pod \"payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt\" (UID: \"6d179607-e322-4187-86d5-e3af6a0b1b60\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt"
Sep 11 12:33:58.898799 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.898734    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt"
Sep 11 12:33:58.927296 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:58.927271    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq"
Sep 11 12:33:59.174026 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:59.173996    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt"]
Sep 11 12:33:59.175742 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:33:59.175712    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d179607_e322_4187_86d5_e3af6a0b1b60.slice/crio-2cbca824da889a864ab25d3d15a4079305fad2c6ec6aeeda1cb57f954c798af3 WatchSource:0}: Error finding container 2cbca824da889a864ab25d3d15a4079305fad2c6ec6aeeda1cb57f954c798af3: Status 404 returned error can't find the container with id 2cbca824da889a864ab25d3d15a4079305fad2c6ec6aeeda1cb57f954c798af3
Sep 11 12:33:59.212005 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:33:59.211945    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq"]
Sep 11 12:33:59.213615 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:33:59.213592    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630ceb15_2cf6_4268_9e8e_27d6c2530fc7.slice/crio-d97f1f233e1738278dc1c9119a89615ae7a4e39887b3c1dc124c6eed3b36add6 WatchSource:0}: Error finding container d97f1f233e1738278dc1c9119a89615ae7a4e39887b3c1dc124c6eed3b36add6: Status 404 returned error can't find the container with id d97f1f233e1738278dc1c9119a89615ae7a4e39887b3c1dc124c6eed3b36add6
Sep 11 12:34:00.137672 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:00.137635    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt" event={"ID":"6d179607-e322-4187-86d5-e3af6a0b1b60","Type":"ContainerStarted","Data":"0d9b0ac847f5f826faedeaf2a3d07855ccef08ac7d8bf5b82f43340e188ca027"}
Sep 11 12:34:00.138174 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:00.137687    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt" event={"ID":"6d179607-e322-4187-86d5-e3af6a0b1b60","Type":"ContainerStarted","Data":"2cbca824da889a864ab25d3d15a4079305fad2c6ec6aeeda1cb57f954c798af3"}
Sep 11 12:34:00.138174 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:00.137738    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt"
Sep 11 12:34:00.139390 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:00.139363    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq" event={"ID":"630ceb15-2cf6-4268-9e8e-27d6c2530fc7","Type":"ContainerStarted","Data":"3bf3d439e0a8420626068caf3e57acee8fea2cc1e408bb8d4eee8e21408799fd"}
Sep 11 12:34:00.139496 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:00.139401    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq" event={"ID":"630ceb15-2cf6-4268-9e8e-27d6c2530fc7","Type":"ContainerStarted","Data":"d97f1f233e1738278dc1c9119a89615ae7a4e39887b3c1dc124c6eed3b36add6"}
Sep 11 12:34:00.139496 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:00.139437    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq"
Sep 11 12:34:00.155133 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:00.155094    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt" podStartSLOduration=2.155082603 podStartE2EDuration="2.155082603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:34:00.153945035 +0000 UTC m=+1171.716724833" watchObservedRunningTime="2026-09-11 12:34:00.155082603 +0000 UTC m=+1171.717862365"
Sep 11 12:34:00.172156 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:00.172118    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq" podStartSLOduration=2.172107272 podStartE2EDuration="2.172107272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:34:00.169435709 +0000 UTC m=+1171.732215484" watchObservedRunningTime="2026-09-11 12:34:00.172107272 +0000 UTC m=+1171.734887049"
Sep 11 12:34:10.218445 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:10.218408    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"]
Sep 11 12:34:10.218892 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:10.218696    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b" podUID="483dc020-9fa7-4bea-98b4-5c65c9adf3e4" containerName="maas-api" containerID="cri-o://1e012e995d6c5411a6b94b44f8d71c1c7463e7efa55d06b1159c5d03e6ff77cf" gracePeriod=30
Sep 11 12:34:10.464316 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:10.464291    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"
Sep 11 12:34:10.597506 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:10.597476    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-openshift-service-ca\") pod \"483dc020-9fa7-4bea-98b4-5c65c9adf3e4\" (UID: \"483dc020-9fa7-4bea-98b4-5c65c9adf3e4\") "
Sep 11 12:34:10.597690 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:10.597589    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-maas-api-tls\") pod \"483dc020-9fa7-4bea-98b4-5c65c9adf3e4\" (UID: \"483dc020-9fa7-4bea-98b4-5c65c9adf3e4\") "
Sep 11 12:34:10.597690 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:10.597621    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-metrics-certs\") pod \"483dc020-9fa7-4bea-98b4-5c65c9adf3e4\" (UID: \"483dc020-9fa7-4bea-98b4-5c65c9adf3e4\") "
Sep 11 12:34:10.597690 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:10.597673    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cttkt\" (UniqueName: \"kubernetes.io/projected/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-kube-api-access-cttkt\") pod \"483dc020-9fa7-4bea-98b4-5c65c9adf3e4\" (UID: \"483dc020-9fa7-4bea-98b4-5c65c9adf3e4\") "
Sep 11 12:34:10.597944 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:10.597914    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "483dc020-9fa7-4bea-98b4-5c65c9adf3e4" (UID: "483dc020-9fa7-4bea-98b4-5c65c9adf3e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:34:10.599758 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:10.599727    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "483dc020-9fa7-4bea-98b4-5c65c9adf3e4" (UID: "483dc020-9fa7-4bea-98b4-5c65c9adf3e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:34:10.599871 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:10.599758    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "483dc020-9fa7-4bea-98b4-5c65c9adf3e4" (UID: "483dc020-9fa7-4bea-98b4-5c65c9adf3e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:34:10.599871 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:10.599820    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-kube-api-access-cttkt" (OuterVolumeSpecName: "kube-api-access-cttkt") pod "483dc020-9fa7-4bea-98b4-5c65c9adf3e4" (UID: "483dc020-9fa7-4bea-98b4-5c65c9adf3e4"). InnerVolumeSpecName "kube-api-access-cttkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:34:10.699247 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:10.699215    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cttkt\" (UniqueName: \"kubernetes.io/projected/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-kube-api-access-cttkt\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:34:10.699247 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:10.699242    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-openshift-service-ca\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:34:10.699247 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:10.699253    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-maas-api-tls\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:34:10.699484 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:10.699262    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/483dc020-9fa7-4bea-98b4-5c65c9adf3e4-metrics-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:34:11.146300 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:11.146268    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq"
Sep 11 12:34:11.146479 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:11.146354    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt"
Sep 11 12:34:11.200345 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:11.200250    2561 generic.go:356] "Generic (PLEG): container finished" podID="483dc020-9fa7-4bea-98b4-5c65c9adf3e4" containerID="1e012e995d6c5411a6b94b44f8d71c1c7463e7efa55d06b1159c5d03e6ff77cf" exitCode=0
Sep 11 12:34:11.200345 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:11.200332    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"
Sep 11 12:34:11.200575 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:11.200326    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b" event={"ID":"483dc020-9fa7-4bea-98b4-5c65c9adf3e4","Type":"ContainerDied","Data":"1e012e995d6c5411a6b94b44f8d71c1c7463e7efa55d06b1159c5d03e6ff77cf"}
Sep 11 12:34:11.200575 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:11.200520    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b" event={"ID":"483dc020-9fa7-4bea-98b4-5c65c9adf3e4","Type":"ContainerDied","Data":"a715aa15b449cc2b6f0649788175b50728f0142222c2a9a2a6eb3fc025ae5778"}
Sep 11 12:34:11.200575 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:11.200567    2561 scope.go:117] "RemoveContainer" containerID="1e012e995d6c5411a6b94b44f8d71c1c7463e7efa55d06b1159c5d03e6ff77cf"
Sep 11 12:34:11.211040 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:11.211017    2561 scope.go:117] "RemoveContainer" containerID="1e012e995d6c5411a6b94b44f8d71c1c7463e7efa55d06b1159c5d03e6ff77cf"
Sep 11 12:34:11.211409 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:34:11.211385    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e012e995d6c5411a6b94b44f8d71c1c7463e7efa55d06b1159c5d03e6ff77cf\": container with ID starting with 1e012e995d6c5411a6b94b44f8d71c1c7463e7efa55d06b1159c5d03e6ff77cf not found: ID does not exist" containerID="1e012e995d6c5411a6b94b44f8d71c1c7463e7efa55d06b1159c5d03e6ff77cf"
Sep 11 12:34:11.211521 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:11.211415    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e012e995d6c5411a6b94b44f8d71c1c7463e7efa55d06b1159c5d03e6ff77cf"} err="failed to get container status \"1e012e995d6c5411a6b94b44f8d71c1c7463e7efa55d06b1159c5d03e6ff77cf\": rpc error: code = NotFound desc = could not find container \"1e012e995d6c5411a6b94b44f8d71c1c7463e7efa55d06b1159c5d03e6ff77cf\": container with ID starting with 1e012e995d6c5411a6b94b44f8d71c1c7463e7efa55d06b1159c5d03e6ff77cf not found: ID does not exist"
Sep 11 12:34:11.220688 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:11.220661    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"]
Sep 11 12:34:11.225317 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:11.225293    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-edffad-6f9d4cfc8-8zk9b"]
Sep 11 12:34:12.931940 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:12.931905    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="483dc020-9fa7-4bea-98b4-5c65c9adf3e4" path="/var/lib/kubelet/pods/483dc020-9fa7-4bea-98b4-5c65c9adf3e4/volumes"
Sep 11 12:34:22.333771 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:22.333737    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-4aaf85"]
Sep 11 12:34:22.334240 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:22.334107    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="483dc020-9fa7-4bea-98b4-5c65c9adf3e4" containerName="maas-api"
Sep 11 12:34:22.334240 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:22.334118    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="483dc020-9fa7-4bea-98b4-5c65c9adf3e4" containerName="maas-api"
Sep 11 12:34:22.334240 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:22.334175    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="483dc020-9fa7-4bea-98b4-5c65c9adf3e4" containerName="maas-api"
Sep 11 12:34:22.338532 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:22.338517    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-4aaf85"
Sep 11 12:34:22.340775 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:22.340756    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h59p7\""
Sep 11 12:34:22.346333 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:22.346314    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-4aaf85"]
Sep 11 12:34:22.504966 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:22.504934    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx2hv\" (UniqueName: \"kubernetes.io/projected/38fc61f5-9aed-48a3-940d-96dd0e36daeb-kube-api-access-sx2hv\") pod \"test-curl-7600-4aaf85\" (UID: \"38fc61f5-9aed-48a3-940d-96dd0e36daeb\") " pod="opendatahub/test-curl-7600-4aaf85"
Sep 11 12:34:22.606478 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:22.606395    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx2hv\" (UniqueName: \"kubernetes.io/projected/38fc61f5-9aed-48a3-940d-96dd0e36daeb-kube-api-access-sx2hv\") pod \"test-curl-7600-4aaf85\" (UID: \"38fc61f5-9aed-48a3-940d-96dd0e36daeb\") " pod="opendatahub/test-curl-7600-4aaf85"
Sep 11 12:34:22.614777 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:22.614748    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx2hv\" (UniqueName: \"kubernetes.io/projected/38fc61f5-9aed-48a3-940d-96dd0e36daeb-kube-api-access-sx2hv\") pod \"test-curl-7600-4aaf85\" (UID: \"38fc61f5-9aed-48a3-940d-96dd0e36daeb\") " pod="opendatahub/test-curl-7600-4aaf85"
Sep 11 12:34:22.626320 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:22.626292    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd"]
Sep 11 12:34:22.629685 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:22.629671    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd"
Sep 11 12:34:22.632157 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:22.632138    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-aaa824-dockercfg-sftcf\""
Sep 11 12:34:22.633151 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:22.633135    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-aaa824\""
Sep 11 12:34:22.640166 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:22.640137    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd"]
Sep 11 12:34:22.647809 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:22.647792    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-4aaf85"
Sep 11 12:34:22.808411 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:22.808380    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppmm6\" (UniqueName: \"kubernetes.io/projected/7e05c25a-76ac-4afd-9472-d8219614eb37-kube-api-access-ppmm6\") pod \"payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd\" (UID: \"7e05c25a-76ac-4afd-9472-d8219614eb37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd"
Sep 11 12:34:22.808594 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:22.808476    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e05c25a-76ac-4afd-9472-d8219614eb37-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd\" (UID: \"7e05c25a-76ac-4afd-9472-d8219614eb37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd"
Sep 11 12:34:22.909925 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:22.909829    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppmm6\" (UniqueName: \"kubernetes.io/projected/7e05c25a-76ac-4afd-9472-d8219614eb37-kube-api-access-ppmm6\") pod \"payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd\" (UID: \"7e05c25a-76ac-4afd-9472-d8219614eb37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd"
Sep 11 12:34:22.910094 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:22.909941    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e05c25a-76ac-4afd-9472-d8219614eb37-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd\" (UID: \"7e05c25a-76ac-4afd-9472-d8219614eb37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd"
Sep 11 12:34:22.910685 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:22.910658    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e05c25a-76ac-4afd-9472-d8219614eb37-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd\" (UID: \"7e05c25a-76ac-4afd-9472-d8219614eb37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd"
Sep 11 12:34:22.918864 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:22.918842    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppmm6\" (UniqueName: \"kubernetes.io/projected/7e05c25a-76ac-4afd-9472-d8219614eb37-kube-api-access-ppmm6\") pod \"payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd\" (UID: \"7e05c25a-76ac-4afd-9472-d8219614eb37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd"
Sep 11 12:34:22.928944 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:34:22.928908    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38fc61f5_9aed_48a3_940d_96dd0e36daeb.slice/crio-f03a1dbc606835f1180fb3b6b30aa701face62eb81bc22702b078872f04227e2 WatchSource:0}: Error finding container f03a1dbc606835f1180fb3b6b30aa701face62eb81bc22702b078872f04227e2: Status 404 returned error can't find the container with id f03a1dbc606835f1180fb3b6b30aa701face62eb81bc22702b078872f04227e2
Sep 11 12:34:22.933237 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:22.933215    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-4aaf85"]
Sep 11 12:34:22.940359 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:22.940337    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd"
Sep 11 12:34:23.227378 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:23.227352    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd"]
Sep 11 12:34:23.229723 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:34:23.229696    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e05c25a_76ac_4afd_9472_d8219614eb37.slice/crio-fc4c2b75c5345bb7479a23ebf823d6ffb2c26c356d45843e795a6409700f7976 WatchSource:0}: Error finding container fc4c2b75c5345bb7479a23ebf823d6ffb2c26c356d45843e795a6409700f7976: Status 404 returned error can't find the container with id fc4c2b75c5345bb7479a23ebf823d6ffb2c26c356d45843e795a6409700f7976
Sep 11 12:34:23.272716 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:23.272687    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd" event={"ID":"7e05c25a-76ac-4afd-9472-d8219614eb37","Type":"ContainerStarted","Data":"fc4c2b75c5345bb7479a23ebf823d6ffb2c26c356d45843e795a6409700f7976"}
Sep 11 12:34:23.274240 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:23.274214    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-4aaf85" event={"ID":"38fc61f5-9aed-48a3-940d-96dd0e36daeb","Type":"ContainerStarted","Data":"8dc21a035b6cd1d71d33dbcd18ecaa3a34de68b8f8685558fe2503dd09304f5e"}
Sep 11 12:34:23.274355 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:23.274252    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-4aaf85" event={"ID":"38fc61f5-9aed-48a3-940d-96dd0e36daeb","Type":"ContainerStarted","Data":"f03a1dbc606835f1180fb3b6b30aa701face62eb81bc22702b078872f04227e2"}
Sep 11 12:34:23.301522 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:23.301474    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7600-4aaf85" podStartSLOduration=1.30145773 podStartE2EDuration="1.30145773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:34:23.300018861 +0000 UTC m=+1194.862798624" watchObservedRunningTime="2026-09-11 12:34:23.30145773 +0000 UTC m=+1194.864237493"
Sep 11 12:34:23.442693 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:23.442657    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:34:23.954708 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:23.954670    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-4aaf85"]
Sep 11 12:34:23.958474 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:23.958443    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-4aaf85"]
Sep 11 12:34:24.079620 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:24.079580    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-dea166"]
Sep 11 12:34:24.080132 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:24.080116    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38fc61f5-9aed-48a3-940d-96dd0e36daeb" containerName="test-curl-7600-4aaf85"
Sep 11 12:34:24.080227 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:24.080134    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="38fc61f5-9aed-48a3-940d-96dd0e36daeb" containerName="test-curl-7600-4aaf85"
Sep 11 12:34:24.080275 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:24.080256    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="38fc61f5-9aed-48a3-940d-96dd0e36daeb" containerName="test-curl-7600-4aaf85"
Sep 11 12:34:24.083875 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:24.083849    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-dea166"
Sep 11 12:34:24.086592 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:24.086569    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-dea166"]
Sep 11 12:34:24.222563 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:24.222528    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lmpt\" (UniqueName: \"kubernetes.io/projected/661b3e51-9acf-4cb3-8087-186d37613b19-kube-api-access-5lmpt\") pod \"test-curl-7600-dea166\" (UID: \"661b3e51-9acf-4cb3-8087-186d37613b19\") " pod="opendatahub/test-curl-7600-dea166"
Sep 11 12:34:24.281385 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:24.281348    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd" event={"ID":"7e05c25a-76ac-4afd-9472-d8219614eb37","Type":"ContainerStarted","Data":"f10a4afeae1dcd58d5d1e358aef498a6b44c285ae15972e0834b3449fd5a491b"}
Sep 11 12:34:24.281570 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:24.281547    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd"
Sep 11 12:34:24.281758 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:34:24.281744    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7600-4aaf85" container="test-curl-7600-4aaf85"
Sep 11 12:34:24.302655 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:24.302596    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd" podStartSLOduration=2.3025753460000002 podStartE2EDuration="2.302575346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:34:24.300182283 +0000 UTC m=+1195.862962048" watchObservedRunningTime="2026-09-11 12:34:24.302575346 +0000 UTC m=+1195.865355110"
Sep 11 12:34:24.323171 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:24.323141    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lmpt\" (UniqueName: \"kubernetes.io/projected/661b3e51-9acf-4cb3-8087-186d37613b19-kube-api-access-5lmpt\") pod \"test-curl-7600-dea166\" (UID: \"661b3e51-9acf-4cb3-8087-186d37613b19\") " pod="opendatahub/test-curl-7600-dea166"
Sep 11 12:34:24.332048 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:24.331984    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lmpt\" (UniqueName: \"kubernetes.io/projected/661b3e51-9acf-4cb3-8087-186d37613b19-kube-api-access-5lmpt\") pod \"test-curl-7600-dea166\" (UID: \"661b3e51-9acf-4cb3-8087-186d37613b19\") " pod="opendatahub/test-curl-7600-dea166"
Sep 11 12:34:24.395118 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:24.395078    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-dea166"
Sep 11 12:34:24.666529 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:24.666502    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-dea166"]
Sep 11 12:34:24.668907 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:34:24.668876    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod661b3e51_9acf_4cb3_8087_186d37613b19.slice/crio-3cde9efeb1041e853032275445e328d2dd5b960758fee29d2bab101094045d58 WatchSource:0}: Error finding container 3cde9efeb1041e853032275445e328d2dd5b960758fee29d2bab101094045d58: Status 404 returned error can't find the container with id 3cde9efeb1041e853032275445e328d2dd5b960758fee29d2bab101094045d58
Sep 11 12:34:25.287702 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:25.287661    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-dea166" event={"ID":"661b3e51-9acf-4cb3-8087-186d37613b19","Type":"ContainerStarted","Data":"1eaf5eb5facd404e75860f85f0e223d444cce162eec557f22b3b57626347d3e4"}
Sep 11 12:34:25.287883 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:25.287710    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-dea166" event={"ID":"661b3e51-9acf-4cb3-8087-186d37613b19","Type":"ContainerStarted","Data":"3cde9efeb1041e853032275445e328d2dd5b960758fee29d2bab101094045d58"}
Sep 11 12:34:25.288004 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:25.287977    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7600-4aaf85" podUID="38fc61f5-9aed-48a3-940d-96dd0e36daeb" containerName="test-curl-7600-4aaf85" containerID="cri-o://8dc21a035b6cd1d71d33dbcd18ecaa3a34de68b8f8685558fe2503dd09304f5e" gracePeriod=2
Sep 11 12:34:25.302318 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:25.302261    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7600-dea166" podStartSLOduration=1.302244959 podStartE2EDuration="1.302244959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:34:25.300888068 +0000 UTC m=+1196.863667834" watchObservedRunningTime="2026-09-11 12:34:25.302244959 +0000 UTC m=+1196.865024722"
Sep 11 12:34:25.940509 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:25.940418    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-dea166"]
Sep 11 12:34:25.947741 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:25.947712    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-dea166"]
Sep 11 12:34:26.292350 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:34:26.292317    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7600-dea166" container="test-curl-7600-dea166"
Sep 11 12:34:27.297298 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:27.297263    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7600-dea166" podUID="661b3e51-9acf-4cb3-8087-186d37613b19" containerName="test-curl-7600-dea166" containerID="cri-o://1eaf5eb5facd404e75860f85f0e223d444cce162eec557f22b3b57626347d3e4" gracePeriod=2
Sep 11 12:34:27.467323 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:27.467294    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-4aaf85"
Sep 11 12:34:27.656453 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:27.656367    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sx2hv\" (UniqueName: \"kubernetes.io/projected/38fc61f5-9aed-48a3-940d-96dd0e36daeb-kube-api-access-sx2hv\") pod \"38fc61f5-9aed-48a3-940d-96dd0e36daeb\" (UID: \"38fc61f5-9aed-48a3-940d-96dd0e36daeb\") "
Sep 11 12:34:27.658501 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:27.658472    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38fc61f5-9aed-48a3-940d-96dd0e36daeb-kube-api-access-sx2hv" (OuterVolumeSpecName: "kube-api-access-sx2hv") pod "38fc61f5-9aed-48a3-940d-96dd0e36daeb" (UID: "38fc61f5-9aed-48a3-940d-96dd0e36daeb"). InnerVolumeSpecName "kube-api-access-sx2hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:34:27.757163 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:27.757126    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx2hv\" (UniqueName: \"kubernetes.io/projected/38fc61f5-9aed-48a3-940d-96dd0e36daeb-kube-api-access-sx2hv\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:34:28.306603 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:28.306571    2561 generic.go:356] "Generic (PLEG): container finished" podID="38fc61f5-9aed-48a3-940d-96dd0e36daeb" containerID="8dc21a035b6cd1d71d33dbcd18ecaa3a34de68b8f8685558fe2503dd09304f5e" exitCode=137
Sep 11 12:34:28.307028 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:28.306619    2561 scope.go:117] "RemoveContainer" containerID="8dc21a035b6cd1d71d33dbcd18ecaa3a34de68b8f8685558fe2503dd09304f5e"
Sep 11 12:34:28.307028 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:28.306615    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-4aaf85"
Sep 11 12:34:28.315122 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:28.315105    2561 scope.go:117] "RemoveContainer" containerID="8dc21a035b6cd1d71d33dbcd18ecaa3a34de68b8f8685558fe2503dd09304f5e"
Sep 11 12:34:28.315427 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:34:28.315407    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc21a035b6cd1d71d33dbcd18ecaa3a34de68b8f8685558fe2503dd09304f5e\": container with ID starting with 8dc21a035b6cd1d71d33dbcd18ecaa3a34de68b8f8685558fe2503dd09304f5e not found: ID does not exist" containerID="8dc21a035b6cd1d71d33dbcd18ecaa3a34de68b8f8685558fe2503dd09304f5e"
Sep 11 12:34:28.315485 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:28.315434    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc21a035b6cd1d71d33dbcd18ecaa3a34de68b8f8685558fe2503dd09304f5e"} err="failed to get container status \"8dc21a035b6cd1d71d33dbcd18ecaa3a34de68b8f8685558fe2503dd09304f5e\": rpc error: code = NotFound desc = could not find container \"8dc21a035b6cd1d71d33dbcd18ecaa3a34de68b8f8685558fe2503dd09304f5e\": container with ID starting with 8dc21a035b6cd1d71d33dbcd18ecaa3a34de68b8f8685558fe2503dd09304f5e not found: ID does not exist"
Sep 11 12:34:28.317909 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:34:28.317882    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7600-4aaf85\" is forbidden: User \"system:node:ip-10-0-131-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-172.ec2.internal' and this object" podUID="38fc61f5-9aed-48a3-940d-96dd0e36daeb" pod="opendatahub/test-curl-7600-4aaf85"
Sep 11 12:34:28.912102 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:28.912073    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcfm6_88402687-4e5b-411c-be7a-4cb955f90425/ovn-acl-logging/0.log"
Sep 11 12:34:28.916270 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:28.916248    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcfm6_88402687-4e5b-411c-be7a-4cb955f90425/ovn-acl-logging/0.log"
Sep 11 12:34:28.931107 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:34:28.931069    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7600-4aaf85\" is forbidden: User \"system:node:ip-10-0-131-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-172.ec2.internal' and this object" podUID="38fc61f5-9aed-48a3-940d-96dd0e36daeb" pod="opendatahub/test-curl-7600-4aaf85"
Sep 11 12:34:28.933842 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:28.933822    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38fc61f5-9aed-48a3-940d-96dd0e36daeb" path="/var/lib/kubelet/pods/38fc61f5-9aed-48a3-940d-96dd0e36daeb/volumes"
Sep 11 12:34:29.480831 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:29.480800    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-dea166"
Sep 11 12:34:29.574770 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:29.574695    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lmpt\" (UniqueName: \"kubernetes.io/projected/661b3e51-9acf-4cb3-8087-186d37613b19-kube-api-access-5lmpt\") pod \"661b3e51-9acf-4cb3-8087-186d37613b19\" (UID: \"661b3e51-9acf-4cb3-8087-186d37613b19\") "
Sep 11 12:34:29.576610 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:29.576587    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661b3e51-9acf-4cb3-8087-186d37613b19-kube-api-access-5lmpt" (OuterVolumeSpecName: "kube-api-access-5lmpt") pod "661b3e51-9acf-4cb3-8087-186d37613b19" (UID: "661b3e51-9acf-4cb3-8087-186d37613b19"). InnerVolumeSpecName "kube-api-access-5lmpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:34:29.675397 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:29.675360    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lmpt\" (UniqueName: \"kubernetes.io/projected/661b3e51-9acf-4cb3-8087-186d37613b19-kube-api-access-5lmpt\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:34:30.319216 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:30.319165    2561 generic.go:356] "Generic (PLEG): container finished" podID="661b3e51-9acf-4cb3-8087-186d37613b19" containerID="1eaf5eb5facd404e75860f85f0e223d444cce162eec557f22b3b57626347d3e4" exitCode=137
Sep 11 12:34:30.319385 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:30.319225    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-dea166"
Sep 11 12:34:30.319385 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:30.319248    2561 scope.go:117] "RemoveContainer" containerID="1eaf5eb5facd404e75860f85f0e223d444cce162eec557f22b3b57626347d3e4"
Sep 11 12:34:30.328363 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:30.328345    2561 scope.go:117] "RemoveContainer" containerID="1eaf5eb5facd404e75860f85f0e223d444cce162eec557f22b3b57626347d3e4"
Sep 11 12:34:30.328622 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:34:30.328603    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eaf5eb5facd404e75860f85f0e223d444cce162eec557f22b3b57626347d3e4\": container with ID starting with 1eaf5eb5facd404e75860f85f0e223d444cce162eec557f22b3b57626347d3e4 not found: ID does not exist" containerID="1eaf5eb5facd404e75860f85f0e223d444cce162eec557f22b3b57626347d3e4"
Sep 11 12:34:30.328686 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:30.328634    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eaf5eb5facd404e75860f85f0e223d444cce162eec557f22b3b57626347d3e4"} err="failed to get container status \"1eaf5eb5facd404e75860f85f0e223d444cce162eec557f22b3b57626347d3e4\": rpc error: code = NotFound desc = could not find container \"1eaf5eb5facd404e75860f85f0e223d444cce162eec557f22b3b57626347d3e4\": container with ID starting with 1eaf5eb5facd404e75860f85f0e223d444cce162eec557f22b3b57626347d3e4 not found: ID does not exist"
Sep 11 12:34:30.329613 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:34:30.329593    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7600-dea166\" is forbidden: User \"system:node:ip-10-0-131-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-172.ec2.internal' and this object" podUID="661b3e51-9acf-4cb3-8087-186d37613b19" pod="opendatahub/test-curl-7600-dea166"
Sep 11 12:34:30.931964 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:30.931929    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="661b3e51-9acf-4cb3-8087-186d37613b19" path="/var/lib/kubelet/pods/661b3e51-9acf-4cb3-8087-186d37613b19/volumes"
Sep 11 12:34:35.289777 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:35.289746    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd"
Sep 11 12:34:36.295455 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.295421    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"]
Sep 11 12:34:36.296004 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.295987    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="661b3e51-9acf-4cb3-8087-186d37613b19" containerName="test-curl-7600-dea166"
Sep 11 12:34:36.296043 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.296007    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="661b3e51-9acf-4cb3-8087-186d37613b19" containerName="test-curl-7600-dea166"
Sep 11 12:34:36.296116 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.296105    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="661b3e51-9acf-4cb3-8087-186d37613b19" containerName="test-curl-7600-dea166"
Sep 11 12:34:36.300586 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.300567    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"
Sep 11 12:34:36.304064 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.304040    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l5tx4\""
Sep 11 12:34:36.304207 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.304096    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 12:34:36.304207 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.304178    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 12:34:36.304319 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.304049    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 12:34:36.304319 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.304260    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-091983a1-serving-cert\""
Sep 11 12:34:36.308068 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.308045    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"]
Sep 11 12:34:36.326401 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.326379    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsfln\" (UniqueName: \"kubernetes.io/projected/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-kube-api-access-gsfln\") pod \"maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8\" (UID: \"c88e12cc-657f-4bfa-bcfe-bf02b20d0603\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"
Sep 11 12:34:36.326507 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.326441    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-maas-api-tls\") pod \"maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8\" (UID: \"c88e12cc-657f-4bfa-bcfe-bf02b20d0603\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"
Sep 11 12:34:36.326507 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.326501    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-metrics-certs\") pod \"maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8\" (UID: \"c88e12cc-657f-4bfa-bcfe-bf02b20d0603\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"
Sep 11 12:34:36.326584 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.326534    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-openshift-service-ca\") pod \"maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8\" (UID: \"c88e12cc-657f-4bfa-bcfe-bf02b20d0603\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"
Sep 11 12:34:36.427243 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.427215    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-maas-api-tls\") pod \"maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8\" (UID: \"c88e12cc-657f-4bfa-bcfe-bf02b20d0603\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"
Sep 11 12:34:36.427370 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.427282    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-metrics-certs\") pod \"maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8\" (UID: \"c88e12cc-657f-4bfa-bcfe-bf02b20d0603\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"
Sep 11 12:34:36.427370 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.427319    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-openshift-service-ca\") pod \"maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8\" (UID: \"c88e12cc-657f-4bfa-bcfe-bf02b20d0603\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"
Sep 11 12:34:36.427457 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:34:36.427379    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-serving-cert: secret "maas-api-e2e-mt-091983a1-serving-cert" not found
Sep 11 12:34:36.427457 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.427387    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsfln\" (UniqueName: \"kubernetes.io/projected/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-kube-api-access-gsfln\") pod \"maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8\" (UID: \"c88e12cc-657f-4bfa-bcfe-bf02b20d0603\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"
Sep 11 12:34:36.427457 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:34:36.427438    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-maas-api-tls podName:c88e12cc-657f-4bfa-bcfe-bf02b20d0603 nodeName:}" failed. No retries permitted until 2026-09-11 12:34:36.927419278 +0000 UTC m=+1208.490199031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-maas-api-tls") pod "maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8" (UID: "c88e12cc-657f-4bfa-bcfe-bf02b20d0603") : secret "maas-api-e2e-mt-091983a1-serving-cert" not found
Sep 11 12:34:36.427986 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.427963    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-openshift-service-ca\") pod \"maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8\" (UID: \"c88e12cc-657f-4bfa-bcfe-bf02b20d0603\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"
Sep 11 12:34:36.429635 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.429613    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-metrics-certs\") pod \"maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8\" (UID: \"c88e12cc-657f-4bfa-bcfe-bf02b20d0603\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"
Sep 11 12:34:36.439104 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.439083    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsfln\" (UniqueName: \"kubernetes.io/projected/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-kube-api-access-gsfln\") pod \"maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8\" (UID: \"c88e12cc-657f-4bfa-bcfe-bf02b20d0603\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"
Sep 11 12:34:36.896706 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.896669    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn"]
Sep 11 12:34:36.900131 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.900116    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn"
Sep 11 12:34:36.902767 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.902744    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-091983a1-dockercfg-fh2b6\""
Sep 11 12:34:36.902999 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.902983    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-091983a1\""
Sep 11 12:34:36.911419 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.911393    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn"]
Sep 11 12:34:36.933612 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.933581    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4302b785-e2d8-4763-8a07-ecc59939a635-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn\" (UID: \"4302b785-e2d8-4763-8a07-ecc59939a635\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn"
Sep 11 12:34:36.933764 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.933736    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbv8z\" (UniqueName: \"kubernetes.io/projected/4302b785-e2d8-4763-8a07-ecc59939a635-kube-api-access-cbv8z\") pod \"payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn\" (UID: \"4302b785-e2d8-4763-8a07-ecc59939a635\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn"
Sep 11 12:34:36.933997 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.933978    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-maas-api-tls\") pod \"maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8\" (UID: \"c88e12cc-657f-4bfa-bcfe-bf02b20d0603\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"
Sep 11 12:34:36.936809 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.936782    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-maas-api-tls\") pod \"maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8\" (UID: \"c88e12cc-657f-4bfa-bcfe-bf02b20d0603\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"
Sep 11 12:34:36.937284 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.937263    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx"]
Sep 11 12:34:36.940577 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.940556    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx"
Sep 11 12:34:36.943926 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:36.943903    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx"]
Sep 11 12:34:37.035089 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:37.035051    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4qkx\" (UniqueName: \"kubernetes.io/projected/cf9abb9f-71b5-42a8-9611-3d5b05cea598-kube-api-access-p4qkx\") pod \"payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx\" (UID: \"cf9abb9f-71b5-42a8-9611-3d5b05cea598\") " pod="openshift-ingress/payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx"
Sep 11 12:34:37.035089 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:37.035091    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbv8z\" (UniqueName: \"kubernetes.io/projected/4302b785-e2d8-4763-8a07-ecc59939a635-kube-api-access-cbv8z\") pod \"payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn\" (UID: \"4302b785-e2d8-4763-8a07-ecc59939a635\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn"
Sep 11 12:34:37.035377 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:37.035149    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9abb9f-71b5-42a8-9611-3d5b05cea598-plugins-config-volume\") pod \"payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx\" (UID: \"cf9abb9f-71b5-42a8-9611-3d5b05cea598\") " pod="openshift-ingress/payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx"
Sep 11 12:34:37.035377 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:37.035210    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4302b785-e2d8-4763-8a07-ecc59939a635-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn\" (UID: \"4302b785-e2d8-4763-8a07-ecc59939a635\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn"
Sep 11 12:34:37.035891 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:37.035869    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4302b785-e2d8-4763-8a07-ecc59939a635-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn\" (UID: \"4302b785-e2d8-4763-8a07-ecc59939a635\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn"
Sep 11 12:34:37.043145 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:37.043125    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbv8z\" (UniqueName: \"kubernetes.io/projected/4302b785-e2d8-4763-8a07-ecc59939a635-kube-api-access-cbv8z\") pod \"payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn\" (UID: \"4302b785-e2d8-4763-8a07-ecc59939a635\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn"
Sep 11 12:34:37.136463 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:37.136423    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9abb9f-71b5-42a8-9611-3d5b05cea598-plugins-config-volume\") pod \"payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx\" (UID: \"cf9abb9f-71b5-42a8-9611-3d5b05cea598\") " pod="openshift-ingress/payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx"
Sep 11 12:34:37.136648 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:37.136552    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4qkx\" (UniqueName: \"kubernetes.io/projected/cf9abb9f-71b5-42a8-9611-3d5b05cea598-kube-api-access-p4qkx\") pod \"payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx\" (UID: \"cf9abb9f-71b5-42a8-9611-3d5b05cea598\") " pod="openshift-ingress/payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx"
Sep 11 12:34:37.137104 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:37.137085    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9abb9f-71b5-42a8-9611-3d5b05cea598-plugins-config-volume\") pod \"payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx\" (UID: \"cf9abb9f-71b5-42a8-9611-3d5b05cea598\") " pod="openshift-ingress/payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx"
Sep 11 12:34:37.149117 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:37.149026    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4qkx\" (UniqueName: \"kubernetes.io/projected/cf9abb9f-71b5-42a8-9611-3d5b05cea598-kube-api-access-p4qkx\") pod \"payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx\" (UID: \"cf9abb9f-71b5-42a8-9611-3d5b05cea598\") " pod="openshift-ingress/payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx"
Sep 11 12:34:37.212003 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:37.211971    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn"
Sep 11 12:34:37.214101 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:37.214074    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"
Sep 11 12:34:37.256493 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:37.256458    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx"
Sep 11 12:34:37.496311 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:37.496259    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"]
Sep 11 12:34:37.498432 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:37.498400    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn"]
Sep 11 12:34:37.499058 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:34:37.499027    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88e12cc_657f_4bfa_bcfe_bf02b20d0603.slice/crio-191471b3f7da3ed312e13eb08fe9698a497907e7cfe7bcac41e3889bfcd0fe1e WatchSource:0}: Error finding container 191471b3f7da3ed312e13eb08fe9698a497907e7cfe7bcac41e3889bfcd0fe1e: Status 404 returned error can't find the container with id 191471b3f7da3ed312e13eb08fe9698a497907e7cfe7bcac41e3889bfcd0fe1e
Sep 11 12:34:37.499498 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:34:37.499472    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4302b785_e2d8_4763_8a07_ecc59939a635.slice/crio-4c9236ed76b644d462b9814941b08fec277e88dad1fc304c8103b13ca6c8d300 WatchSource:0}: Error finding container 4c9236ed76b644d462b9814941b08fec277e88dad1fc304c8103b13ca6c8d300: Status 404 returned error can't find the container with id 4c9236ed76b644d462b9814941b08fec277e88dad1fc304c8103b13ca6c8d300
Sep 11 12:34:37.500663 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:37.500643    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 12:34:37.553001 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:37.552974    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx"]
Sep 11 12:34:37.554579 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:34:37.554556    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9abb9f_71b5_42a8_9611_3d5b05cea598.slice/crio-72691c8092c639df93043ef7d98bea078a80bb4bac5af90af5cd9f88330d4232 WatchSource:0}: Error finding container 72691c8092c639df93043ef7d98bea078a80bb4bac5af90af5cd9f88330d4232: Status 404 returned error can't find the container with id 72691c8092c639df93043ef7d98bea078a80bb4bac5af90af5cd9f88330d4232
Sep 11 12:34:38.359980 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:38.359947    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8" event={"ID":"c88e12cc-657f-4bfa-bcfe-bf02b20d0603","Type":"ContainerStarted","Data":"78c0db6e75cb52b91269ffa1130844f5b2ea92ff1ff022f94e355bff72526885"}
Sep 11 12:34:38.360213 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:38.359999    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8" event={"ID":"c88e12cc-657f-4bfa-bcfe-bf02b20d0603","Type":"ContainerStarted","Data":"191471b3f7da3ed312e13eb08fe9698a497907e7cfe7bcac41e3889bfcd0fe1e"}
Sep 11 12:34:38.360213 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:38.360048    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"
Sep 11 12:34:38.361545 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:38.361520    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx" event={"ID":"cf9abb9f-71b5-42a8-9611-3d5b05cea598","Type":"ContainerStarted","Data":"bbd722bda9d6bb7dde303c778d2b4b21caf78771ec1f4d363d0a6e06e38d6b87"}
Sep 11 12:34:38.361634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:38.361572    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx"
Sep 11 12:34:38.361634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:38.361591    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx" event={"ID":"cf9abb9f-71b5-42a8-9611-3d5b05cea598","Type":"ContainerStarted","Data":"72691c8092c639df93043ef7d98bea078a80bb4bac5af90af5cd9f88330d4232"}
Sep 11 12:34:38.362968 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:38.362948    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn" event={"ID":"4302b785-e2d8-4763-8a07-ecc59939a635","Type":"ContainerStarted","Data":"34ef0b75c542e283f0bd2aecb04ac1a58e6e70e6687e6fd7da4b6aa74f21b2e4"}
Sep 11 12:34:38.363046 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:38.362981    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn" event={"ID":"4302b785-e2d8-4763-8a07-ecc59939a635","Type":"ContainerStarted","Data":"4c9236ed76b644d462b9814941b08fec277e88dad1fc304c8103b13ca6c8d300"}
Sep 11 12:34:38.363046 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:38.363025    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn"
Sep 11 12:34:38.379839 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:38.379798    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8" podStartSLOduration=1.986474607 podStartE2EDuration="2.379785635s" totalImagesPullingTime="393.311028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:34:38.378835602 +0000 UTC m=+1209.941615366" watchObservedRunningTime="2026-09-11 12:34:38.379785635 +0000 UTC m=+1209.942565399"
Sep 11 12:34:38.398820 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:38.398775    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn" podStartSLOduration=2.398762838 podStartE2EDuration="2.398762838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:34:38.397176883 +0000 UTC m=+1209.959956647" watchObservedRunningTime="2026-09-11 12:34:38.398762838 +0000 UTC m=+1209.961542600"
Sep 11 12:34:38.415201 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:38.415157    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx" podStartSLOduration=2.4151437 podStartE2EDuration="2.4151437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:34:38.413065878 +0000 UTC m=+1209.975845642" watchObservedRunningTime="2026-09-11 12:34:38.4151437 +0000 UTC m=+1209.977923462"
Sep 11 12:34:44.374561 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:44.374527    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"
Sep 11 12:34:48.330443 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:48.330406    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:34:49.371374 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:49.371334    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn"
Sep 11 12:34:49.371868 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:49.371559    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx"
Sep 11 12:34:51.090296 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:51.090256    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n"]
Sep 11 12:34:51.093977 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:51.093955    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n"
Sep 11 12:34:51.096400 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:51.096378    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-f4502d-dockercfg-ppw6l\""
Sep 11 12:34:51.096501 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:51.096378    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-f4502d\""
Sep 11 12:34:51.104386 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:51.104364    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n"]
Sep 11 12:34:51.171600 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:51.171558    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jd7k\" (UniqueName: \"kubernetes.io/projected/260e8e9e-7393-4be6-b6e7-57a5a1512f40-kube-api-access-8jd7k\") pod \"payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n\" (UID: \"260e8e9e-7393-4be6-b6e7-57a5a1512f40\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n"
Sep 11 12:34:51.171738 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:51.171675    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260e8e9e-7393-4be6-b6e7-57a5a1512f40-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n\" (UID: \"260e8e9e-7393-4be6-b6e7-57a5a1512f40\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n"
Sep 11 12:34:51.272989 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:51.272944    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260e8e9e-7393-4be6-b6e7-57a5a1512f40-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n\" (UID: \"260e8e9e-7393-4be6-b6e7-57a5a1512f40\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n"
Sep 11 12:34:51.273159 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:51.273067    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jd7k\" (UniqueName: \"kubernetes.io/projected/260e8e9e-7393-4be6-b6e7-57a5a1512f40-kube-api-access-8jd7k\") pod \"payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n\" (UID: \"260e8e9e-7393-4be6-b6e7-57a5a1512f40\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n"
Sep 11 12:34:51.273687 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:51.273664    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260e8e9e-7393-4be6-b6e7-57a5a1512f40-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n\" (UID: \"260e8e9e-7393-4be6-b6e7-57a5a1512f40\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n"
Sep 11 12:34:51.281328 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:51.281306    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jd7k\" (UniqueName: \"kubernetes.io/projected/260e8e9e-7393-4be6-b6e7-57a5a1512f40-kube-api-access-8jd7k\") pod \"payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n\" (UID: \"260e8e9e-7393-4be6-b6e7-57a5a1512f40\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n"
Sep 11 12:34:51.406120 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:51.406036    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n"
Sep 11 12:34:51.680340 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:51.680260    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n"]
Sep 11 12:34:51.683844 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:34:51.683820    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260e8e9e_7393_4be6_b6e7_57a5a1512f40.slice/crio-759f7353f1f451d84aa22c16cb318b9f25c12584ceef6a79979013a55efbd1cd WatchSource:0}: Error finding container 759f7353f1f451d84aa22c16cb318b9f25c12584ceef6a79979013a55efbd1cd: Status 404 returned error can't find the container with id 759f7353f1f451d84aa22c16cb318b9f25c12584ceef6a79979013a55efbd1cd
Sep 11 12:34:52.435437 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:52.435398    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n" event={"ID":"260e8e9e-7393-4be6-b6e7-57a5a1512f40","Type":"ContainerStarted","Data":"79f8ba0480c196dea91ce66c8a6eb6bfa8281e63591a6932f3e64be13584ce01"}
Sep 11 12:34:52.435826 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:52.435495    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n"
Sep 11 12:34:52.435826 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:52.435523    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n" event={"ID":"260e8e9e-7393-4be6-b6e7-57a5a1512f40","Type":"ContainerStarted","Data":"759f7353f1f451d84aa22c16cb318b9f25c12584ceef6a79979013a55efbd1cd"}
Sep 11 12:34:52.453081 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:34:52.453033    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n" podStartSLOduration=1.45301862 podStartE2EDuration="1.45301862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:34:52.451156868 +0000 UTC m=+1224.013936631" watchObservedRunningTime="2026-09-11 12:34:52.45301862 +0000 UTC m=+1224.015798382"
Sep 11 12:35:03.442873 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:03.442842    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n"
Sep 11 12:35:27.340395 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:27.340360    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:35:38.877557 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:38.877523    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"]
Sep 11 12:35:53.896886 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:53.896846    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr" podUID="630eedab-4510-426b-9735-f9b976b5964c" containerName="main" containerID="cri-o://e50c72467019a3feeab10c0ec01f9a668e5c139b8de062d7ed71b0329b88a24c" gracePeriod=45
Sep 11 12:35:54.152166 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.152111    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:35:54.266802 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.266771    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630eedab-4510-426b-9735-f9b976b5964c-tls-certs\") pod \"630eedab-4510-426b-9735-f9b976b5964c\" (UID: \"630eedab-4510-426b-9735-f9b976b5964c\") "
Sep 11 12:35:54.266969 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.266871    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/630eedab-4510-426b-9735-f9b976b5964c-home\") pod \"630eedab-4510-426b-9735-f9b976b5964c\" (UID: \"630eedab-4510-426b-9735-f9b976b5964c\") "
Sep 11 12:35:54.266969 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.266926    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/630eedab-4510-426b-9735-f9b976b5964c-dshm\") pod \"630eedab-4510-426b-9735-f9b976b5964c\" (UID: \"630eedab-4510-426b-9735-f9b976b5964c\") "
Sep 11 12:35:54.267080 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.266980    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5gn2\" (UniqueName: \"kubernetes.io/projected/630eedab-4510-426b-9735-f9b976b5964c-kube-api-access-v5gn2\") pod \"630eedab-4510-426b-9735-f9b976b5964c\" (UID: \"630eedab-4510-426b-9735-f9b976b5964c\") "
Sep 11 12:35:54.267080 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.267006    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/630eedab-4510-426b-9735-f9b976b5964c-model-cache\") pod \"630eedab-4510-426b-9735-f9b976b5964c\" (UID: \"630eedab-4510-426b-9735-f9b976b5964c\") "
Sep 11 12:35:54.267080 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.267037    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630eedab-4510-426b-9735-f9b976b5964c-tmp-dir\") pod \"630eedab-4510-426b-9735-f9b976b5964c\" (UID: \"630eedab-4510-426b-9735-f9b976b5964c\") "
Sep 11 12:35:54.267273 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.267149    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630eedab-4510-426b-9735-f9b976b5964c-home" (OuterVolumeSpecName: "home") pod "630eedab-4510-426b-9735-f9b976b5964c" (UID: "630eedab-4510-426b-9735-f9b976b5964c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:35:54.267339 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.267293    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630eedab-4510-426b-9735-f9b976b5964c-model-cache" (OuterVolumeSpecName: "model-cache") pod "630eedab-4510-426b-9735-f9b976b5964c" (UID: "630eedab-4510-426b-9735-f9b976b5964c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:35:54.267393 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.267349    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630eedab-4510-426b-9735-f9b976b5964c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "630eedab-4510-426b-9735-f9b976b5964c" (UID: "630eedab-4510-426b-9735-f9b976b5964c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:35:54.267476 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.267460    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/630eedab-4510-426b-9735-f9b976b5964c-model-cache\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:35:54.267521 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.267479    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630eedab-4510-426b-9735-f9b976b5964c-tmp-dir\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:35:54.267521 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.267488    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/630eedab-4510-426b-9735-f9b976b5964c-home\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:35:54.269135 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.269105    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630eedab-4510-426b-9735-f9b976b5964c-kube-api-access-v5gn2" (OuterVolumeSpecName: "kube-api-access-v5gn2") pod "630eedab-4510-426b-9735-f9b976b5964c" (UID: "630eedab-4510-426b-9735-f9b976b5964c"). InnerVolumeSpecName "kube-api-access-v5gn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:35:54.269396 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.269376    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630eedab-4510-426b-9735-f9b976b5964c-dshm" (OuterVolumeSpecName: "dshm") pod "630eedab-4510-426b-9735-f9b976b5964c" (UID: "630eedab-4510-426b-9735-f9b976b5964c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:35:54.269473 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.269392    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/630eedab-4510-426b-9735-f9b976b5964c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "630eedab-4510-426b-9735-f9b976b5964c" (UID: "630eedab-4510-426b-9735-f9b976b5964c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:35:54.369054 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.369024    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/630eedab-4510-426b-9735-f9b976b5964c-dshm\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:35:54.369054 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.369054    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5gn2\" (UniqueName: \"kubernetes.io/projected/630eedab-4510-426b-9735-f9b976b5964c-kube-api-access-v5gn2\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:35:54.369227 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.369069    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630eedab-4510-426b-9735-f9b976b5964c-tls-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:35:54.763611 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.763575    2561 generic.go:356] "Generic (PLEG): container finished" podID="630eedab-4510-426b-9735-f9b976b5964c" containerID="e50c72467019a3feeab10c0ec01f9a668e5c139b8de062d7ed71b0329b88a24c" exitCode=0
Sep 11 12:35:54.763787 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.763620    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"
Sep 11 12:35:54.763787 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.763673    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr" event={"ID":"630eedab-4510-426b-9735-f9b976b5964c","Type":"ContainerDied","Data":"e50c72467019a3feeab10c0ec01f9a668e5c139b8de062d7ed71b0329b88a24c"}
Sep 11 12:35:54.763787 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.763717    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr" event={"ID":"630eedab-4510-426b-9735-f9b976b5964c","Type":"ContainerDied","Data":"6464a56048606c22ff57c5871cac9c4cb709b0f60b4fe4269296ba763ea03024"}
Sep 11 12:35:54.763787 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.763735    2561 scope.go:117] "RemoveContainer" containerID="e50c72467019a3feeab10c0ec01f9a668e5c139b8de062d7ed71b0329b88a24c"
Sep 11 12:35:54.772629 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.772609    2561 scope.go:117] "RemoveContainer" containerID="e50c72467019a3feeab10c0ec01f9a668e5c139b8de062d7ed71b0329b88a24c"
Sep 11 12:35:54.772922 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:35:54.772897    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50c72467019a3feeab10c0ec01f9a668e5c139b8de062d7ed71b0329b88a24c\": container with ID starting with e50c72467019a3feeab10c0ec01f9a668e5c139b8de062d7ed71b0329b88a24c not found: ID does not exist" containerID="e50c72467019a3feeab10c0ec01f9a668e5c139b8de062d7ed71b0329b88a24c"
Sep 11 12:35:54.772968 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.772935    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50c72467019a3feeab10c0ec01f9a668e5c139b8de062d7ed71b0329b88a24c"} err="failed to get container status \"e50c72467019a3feeab10c0ec01f9a668e5c139b8de062d7ed71b0329b88a24c\": rpc error: code = NotFound desc = could not find container \"e50c72467019a3feeab10c0ec01f9a668e5c139b8de062d7ed71b0329b88a24c\": container with ID starting with e50c72467019a3feeab10c0ec01f9a668e5c139b8de062d7ed71b0329b88a24c not found: ID does not exist"
Sep 11 12:35:54.784183 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.784162    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"]
Sep 11 12:35:54.788052 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.788027    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-a-97e93ffc-kserve-59c5d9b489-mndpr"]
Sep 11 12:35:54.933438 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:35:54.933399    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="630eedab-4510-426b-9735-f9b976b5964c" path="/var/lib/kubelet/pods/630eedab-4510-426b-9735-f9b976b5964c/volumes"
Sep 11 12:36:05.554500 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:05.554462    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:36:26.689324 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:26.689289    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"]
Sep 11 12:36:26.689935 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:26.689573    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8" podUID="c88e12cc-657f-4bfa-bcfe-bf02b20d0603" containerName="maas-api" containerID="cri-o://78c0db6e75cb52b91269ffa1130844f5b2ea92ff1ff022f94e355bff72526885" gracePeriod=30
Sep 11 12:36:26.752089 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:26.752059    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx"]
Sep 11 12:36:26.752311 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:26.752284    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx" podUID="cf9abb9f-71b5-42a8-9611-3d5b05cea598" containerName="payload-processing" containerID="cri-o://bbd722bda9d6bb7dde303c778d2b4b21caf78771ec1f4d363d0a6e06e38d6b87" gracePeriod=30
Sep 11 12:36:26.759008 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:26.758986    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn"]
Sep 11 12:36:26.759403 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:26.759228    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn" podUID="4302b785-e2d8-4763-8a07-ecc59939a635" containerName="payload-pre-processing" containerID="cri-o://34ef0b75c542e283f0bd2aecb04ac1a58e6e70e6687e6fd7da4b6aa74f21b2e4" gracePeriod=30
Sep 11 12:36:26.938365 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:26.938329    2561 generic.go:356] "Generic (PLEG): container finished" podID="4302b785-e2d8-4763-8a07-ecc59939a635" containerID="34ef0b75c542e283f0bd2aecb04ac1a58e6e70e6687e6fd7da4b6aa74f21b2e4" exitCode=0
Sep 11 12:36:26.938524 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:26.938379    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn" event={"ID":"4302b785-e2d8-4763-8a07-ecc59939a635","Type":"ContainerDied","Data":"34ef0b75c542e283f0bd2aecb04ac1a58e6e70e6687e6fd7da4b6aa74f21b2e4"}
Sep 11 12:36:26.940601 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:26.940537    2561 generic.go:356] "Generic (PLEG): container finished" podID="c88e12cc-657f-4bfa-bcfe-bf02b20d0603" containerID="78c0db6e75cb52b91269ffa1130844f5b2ea92ff1ff022f94e355bff72526885" exitCode=0
Sep 11 12:36:26.940717 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:26.940638    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8" event={"ID":"c88e12cc-657f-4bfa-bcfe-bf02b20d0603","Type":"ContainerDied","Data":"78c0db6e75cb52b91269ffa1130844f5b2ea92ff1ff022f94e355bff72526885"}
Sep 11 12:36:26.942623 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:26.942603    2561 generic.go:356] "Generic (PLEG): container finished" podID="cf9abb9f-71b5-42a8-9611-3d5b05cea598" containerID="bbd722bda9d6bb7dde303c778d2b4b21caf78771ec1f4d363d0a6e06e38d6b87" exitCode=0
Sep 11 12:36:26.942754 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:26.942629    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx" event={"ID":"cf9abb9f-71b5-42a8-9611-3d5b05cea598","Type":"ContainerDied","Data":"bbd722bda9d6bb7dde303c778d2b4b21caf78771ec1f4d363d0a6e06e38d6b87"}
Sep 11 12:36:27.005859 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.005836    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx"
Sep 11 12:36:27.023080 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.023059    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"
Sep 11 12:36:27.063709 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.063686    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn"
Sep 11 12:36:27.173065 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.173032    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsfln\" (UniqueName: \"kubernetes.io/projected/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-kube-api-access-gsfln\") pod \"c88e12cc-657f-4bfa-bcfe-bf02b20d0603\" (UID: \"c88e12cc-657f-4bfa-bcfe-bf02b20d0603\") "
Sep 11 12:36:27.173260 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.173106    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-metrics-certs\") pod \"c88e12cc-657f-4bfa-bcfe-bf02b20d0603\" (UID: \"c88e12cc-657f-4bfa-bcfe-bf02b20d0603\") "
Sep 11 12:36:27.173260 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.173146    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4qkx\" (UniqueName: \"kubernetes.io/projected/cf9abb9f-71b5-42a8-9611-3d5b05cea598-kube-api-access-p4qkx\") pod \"cf9abb9f-71b5-42a8-9611-3d5b05cea598\" (UID: \"cf9abb9f-71b5-42a8-9611-3d5b05cea598\") "
Sep 11 12:36:27.173260 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.173168    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9abb9f-71b5-42a8-9611-3d5b05cea598-plugins-config-volume\") pod \"cf9abb9f-71b5-42a8-9611-3d5b05cea598\" (UID: \"cf9abb9f-71b5-42a8-9611-3d5b05cea598\") "
Sep 11 12:36:27.173260 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.173230    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-openshift-service-ca\") pod \"c88e12cc-657f-4bfa-bcfe-bf02b20d0603\" (UID: \"c88e12cc-657f-4bfa-bcfe-bf02b20d0603\") "
Sep 11 12:36:27.173500 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.173284    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbv8z\" (UniqueName: \"kubernetes.io/projected/4302b785-e2d8-4763-8a07-ecc59939a635-kube-api-access-cbv8z\") pod \"4302b785-e2d8-4763-8a07-ecc59939a635\" (UID: \"4302b785-e2d8-4763-8a07-ecc59939a635\") "
Sep 11 12:36:27.173500 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.173324    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-maas-api-tls\") pod \"c88e12cc-657f-4bfa-bcfe-bf02b20d0603\" (UID: \"c88e12cc-657f-4bfa-bcfe-bf02b20d0603\") "
Sep 11 12:36:27.173500 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.173366    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4302b785-e2d8-4763-8a07-ecc59939a635-plugins-config-volume\") pod \"4302b785-e2d8-4763-8a07-ecc59939a635\" (UID: \"4302b785-e2d8-4763-8a07-ecc59939a635\") "
Sep 11 12:36:27.173658 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.173632    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9abb9f-71b5-42a8-9611-3d5b05cea598-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cf9abb9f-71b5-42a8-9611-3d5b05cea598" (UID: "cf9abb9f-71b5-42a8-9611-3d5b05cea598"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:36:27.173658 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.173630    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c88e12cc-657f-4bfa-bcfe-bf02b20d0603" (UID: "c88e12cc-657f-4bfa-bcfe-bf02b20d0603"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:36:27.173883 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.173837    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9abb9f-71b5-42a8-9611-3d5b05cea598-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:36:27.173883 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.173863    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-openshift-service-ca\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:36:27.174006 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.173976    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4302b785-e2d8-4763-8a07-ecc59939a635-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4302b785-e2d8-4763-8a07-ecc59939a635" (UID: "4302b785-e2d8-4763-8a07-ecc59939a635"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:36:27.175509 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.175478    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-kube-api-access-gsfln" (OuterVolumeSpecName: "kube-api-access-gsfln") pod "c88e12cc-657f-4bfa-bcfe-bf02b20d0603" (UID: "c88e12cc-657f-4bfa-bcfe-bf02b20d0603"). InnerVolumeSpecName "kube-api-access-gsfln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:36:27.175623 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.175515    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c88e12cc-657f-4bfa-bcfe-bf02b20d0603" (UID: "c88e12cc-657f-4bfa-bcfe-bf02b20d0603"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:36:27.175791 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.175759    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4302b785-e2d8-4763-8a07-ecc59939a635-kube-api-access-cbv8z" (OuterVolumeSpecName: "kube-api-access-cbv8z") pod "4302b785-e2d8-4763-8a07-ecc59939a635" (UID: "4302b785-e2d8-4763-8a07-ecc59939a635"). InnerVolumeSpecName "kube-api-access-cbv8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:36:27.175791 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.175773    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c88e12cc-657f-4bfa-bcfe-bf02b20d0603" (UID: "c88e12cc-657f-4bfa-bcfe-bf02b20d0603"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:36:27.175791 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.175772    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9abb9f-71b5-42a8-9611-3d5b05cea598-kube-api-access-p4qkx" (OuterVolumeSpecName: "kube-api-access-p4qkx") pod "cf9abb9f-71b5-42a8-9611-3d5b05cea598" (UID: "cf9abb9f-71b5-42a8-9611-3d5b05cea598"). InnerVolumeSpecName "kube-api-access-p4qkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:36:27.274904 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.274868    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbv8z\" (UniqueName: \"kubernetes.io/projected/4302b785-e2d8-4763-8a07-ecc59939a635-kube-api-access-cbv8z\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:36:27.274904 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.274900    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-maas-api-tls\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:36:27.275099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.274915    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4302b785-e2d8-4763-8a07-ecc59939a635-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:36:27.275099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.274928    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsfln\" (UniqueName: \"kubernetes.io/projected/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-kube-api-access-gsfln\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:36:27.275099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.274938    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c88e12cc-657f-4bfa-bcfe-bf02b20d0603-metrics-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:36:27.275099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.274950    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4qkx\" (UniqueName: \"kubernetes.io/projected/cf9abb9f-71b5-42a8-9611-3d5b05cea598-kube-api-access-p4qkx\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:36:27.949541 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.949506    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"
Sep 11 12:36:27.949983 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.949495    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8" event={"ID":"c88e12cc-657f-4bfa-bcfe-bf02b20d0603","Type":"ContainerDied","Data":"191471b3f7da3ed312e13eb08fe9698a497907e7cfe7bcac41e3889bfcd0fe1e"}
Sep 11 12:36:27.949983 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.949644    2561 scope.go:117] "RemoveContainer" containerID="78c0db6e75cb52b91269ffa1130844f5b2ea92ff1ff022f94e355bff72526885"
Sep 11 12:36:27.951126 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.950941    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx" event={"ID":"cf9abb9f-71b5-42a8-9611-3d5b05cea598","Type":"ContainerDied","Data":"72691c8092c639df93043ef7d98bea078a80bb4bac5af90af5cd9f88330d4232"}
Sep 11 12:36:27.951126 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.950964    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx"
Sep 11 12:36:27.952538 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.952510    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn" event={"ID":"4302b785-e2d8-4763-8a07-ecc59939a635","Type":"ContainerDied","Data":"4c9236ed76b644d462b9814941b08fec277e88dad1fc304c8103b13ca6c8d300"}
Sep 11 12:36:27.952538 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.952537    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn"
Sep 11 12:36:27.960860 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.960843    2561 scope.go:117] "RemoveContainer" containerID="bbd722bda9d6bb7dde303c778d2b4b21caf78771ec1f4d363d0a6e06e38d6b87"
Sep 11 12:36:27.969205 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.969170    2561 scope.go:117] "RemoveContainer" containerID="34ef0b75c542e283f0bd2aecb04ac1a58e6e70e6687e6fd7da4b6aa74f21b2e4"
Sep 11 12:36:27.980268 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.980245    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"]
Sep 11 12:36:27.982106 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.982084    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-091983a1-7d5c5ffff9-gt9l8"]
Sep 11 12:36:27.991640 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.991615    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn"]
Sep 11 12:36:27.995458 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:27.995437    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-091983a1-7f5bc49f5d-695cn"]
Sep 11 12:36:28.004955 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:28.004935    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx"]
Sep 11 12:36:28.011073 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:28.011049    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-091983a1-6d9b445f85-zdkhx"]
Sep 11 12:36:28.933352 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:28.933323    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4302b785-e2d8-4763-8a07-ecc59939a635" path="/var/lib/kubelet/pods/4302b785-e2d8-4763-8a07-ecc59939a635/volumes"
Sep 11 12:36:28.933738 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:28.933725    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c88e12cc-657f-4bfa-bcfe-bf02b20d0603" path="/var/lib/kubelet/pods/c88e12cc-657f-4bfa-bcfe-bf02b20d0603/volumes"
Sep 11 12:36:28.934046 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:28.934034    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf9abb9f-71b5-42a8-9611-3d5b05cea598" path="/var/lib/kubelet/pods/cf9abb9f-71b5-42a8-9611-3d5b05cea598/volumes"
Sep 11 12:36:36.432489 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:36.432447    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:36:54.790993 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.790914    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"]
Sep 11 12:36:54.791345 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.791337    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="630eedab-4510-426b-9735-f9b976b5964c" containerName="main"
Sep 11 12:36:54.791393 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.791350    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="630eedab-4510-426b-9735-f9b976b5964c" containerName="main"
Sep 11 12:36:54.791393 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.791360    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf9abb9f-71b5-42a8-9611-3d5b05cea598" containerName="payload-processing"
Sep 11 12:36:54.791393 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.791366    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf9abb9f-71b5-42a8-9611-3d5b05cea598" containerName="payload-processing"
Sep 11 12:36:54.791393 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.791385    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c88e12cc-657f-4bfa-bcfe-bf02b20d0603" containerName="maas-api"
Sep 11 12:36:54.791393 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.791390    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c88e12cc-657f-4bfa-bcfe-bf02b20d0603" containerName="maas-api"
Sep 11 12:36:54.791551 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.791405    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4302b785-e2d8-4763-8a07-ecc59939a635" containerName="payload-pre-processing"
Sep 11 12:36:54.791551 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.791411    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4302b785-e2d8-4763-8a07-ecc59939a635" containerName="payload-pre-processing"
Sep 11 12:36:54.791551 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.791468    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4302b785-e2d8-4763-8a07-ecc59939a635" containerName="payload-pre-processing"
Sep 11 12:36:54.791551 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.791476    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf9abb9f-71b5-42a8-9611-3d5b05cea598" containerName="payload-processing"
Sep 11 12:36:54.791551 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.791482    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="630eedab-4510-426b-9735-f9b976b5964c" containerName="main"
Sep 11 12:36:54.791551 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.791487    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c88e12cc-657f-4bfa-bcfe-bf02b20d0603" containerName="maas-api"
Sep 11 12:36:54.796181 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.796162    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:36:54.799653 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.799630    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4502d\"/\"kube-root-ca.crt\""
Sep 11 12:36:54.799839 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.799815    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4502d\"/\"auth-test-model-f4502d-kserve-self-signed-certs\""
Sep 11 12:36:54.799914 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.799887    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4502d\"/\"default-dockercfg-p8khr\""
Sep 11 12:36:54.800015 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.799993    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4502d\"/\"openshift-service-ca.crt\""
Sep 11 12:36:54.811249 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.811226    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"]
Sep 11 12:36:54.918039 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.918006    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15013cc9-0933-40e0-958e-bc31edd1f3bf-home\") pod \"auth-test-model-f4502d-kserve-7944f5f479-zfv22\" (UID: \"15013cc9-0933-40e0-958e-bc31edd1f3bf\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:36:54.918039 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.918040    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15013cc9-0933-40e0-958e-bc31edd1f3bf-tls-certs\") pod \"auth-test-model-f4502d-kserve-7944f5f479-zfv22\" (UID: \"15013cc9-0933-40e0-958e-bc31edd1f3bf\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:36:54.918308 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.918063    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15013cc9-0933-40e0-958e-bc31edd1f3bf-tmp-dir\") pod \"auth-test-model-f4502d-kserve-7944f5f479-zfv22\" (UID: \"15013cc9-0933-40e0-958e-bc31edd1f3bf\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:36:54.918308 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.918123    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15013cc9-0933-40e0-958e-bc31edd1f3bf-dshm\") pod \"auth-test-model-f4502d-kserve-7944f5f479-zfv22\" (UID: \"15013cc9-0933-40e0-958e-bc31edd1f3bf\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:36:54.918308 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.918178    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klhc2\" (UniqueName: \"kubernetes.io/projected/15013cc9-0933-40e0-958e-bc31edd1f3bf-kube-api-access-klhc2\") pod \"auth-test-model-f4502d-kserve-7944f5f479-zfv22\" (UID: \"15013cc9-0933-40e0-958e-bc31edd1f3bf\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:36:54.918308 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:54.918263    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15013cc9-0933-40e0-958e-bc31edd1f3bf-model-cache\") pod \"auth-test-model-f4502d-kserve-7944f5f479-zfv22\" (UID: \"15013cc9-0933-40e0-958e-bc31edd1f3bf\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:36:55.018951 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:55.018915    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klhc2\" (UniqueName: \"kubernetes.io/projected/15013cc9-0933-40e0-958e-bc31edd1f3bf-kube-api-access-klhc2\") pod \"auth-test-model-f4502d-kserve-7944f5f479-zfv22\" (UID: \"15013cc9-0933-40e0-958e-bc31edd1f3bf\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:36:55.019149 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:55.018961    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15013cc9-0933-40e0-958e-bc31edd1f3bf-model-cache\") pod \"auth-test-model-f4502d-kserve-7944f5f479-zfv22\" (UID: \"15013cc9-0933-40e0-958e-bc31edd1f3bf\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:36:55.019149 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:55.019107    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15013cc9-0933-40e0-958e-bc31edd1f3bf-home\") pod \"auth-test-model-f4502d-kserve-7944f5f479-zfv22\" (UID: \"15013cc9-0933-40e0-958e-bc31edd1f3bf\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:36:55.019149 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:55.019132    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15013cc9-0933-40e0-958e-bc31edd1f3bf-tls-certs\") pod \"auth-test-model-f4502d-kserve-7944f5f479-zfv22\" (UID: \"15013cc9-0933-40e0-958e-bc31edd1f3bf\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:36:55.019396 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:55.019172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15013cc9-0933-40e0-958e-bc31edd1f3bf-tmp-dir\") pod \"auth-test-model-f4502d-kserve-7944f5f479-zfv22\" (UID: \"15013cc9-0933-40e0-958e-bc31edd1f3bf\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:36:55.019396 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:55.019230    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15013cc9-0933-40e0-958e-bc31edd1f3bf-dshm\") pod \"auth-test-model-f4502d-kserve-7944f5f479-zfv22\" (UID: \"15013cc9-0933-40e0-958e-bc31edd1f3bf\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:36:55.019506 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:55.019413    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15013cc9-0933-40e0-958e-bc31edd1f3bf-model-cache\") pod \"auth-test-model-f4502d-kserve-7944f5f479-zfv22\" (UID: \"15013cc9-0933-40e0-958e-bc31edd1f3bf\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:36:55.019506 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:55.019469    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15013cc9-0933-40e0-958e-bc31edd1f3bf-home\") pod \"auth-test-model-f4502d-kserve-7944f5f479-zfv22\" (UID: \"15013cc9-0933-40e0-958e-bc31edd1f3bf\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:36:55.019620 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:55.019566    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15013cc9-0933-40e0-958e-bc31edd1f3bf-tmp-dir\") pod \"auth-test-model-f4502d-kserve-7944f5f479-zfv22\" (UID: \"15013cc9-0933-40e0-958e-bc31edd1f3bf\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:36:55.021469 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:55.021441    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15013cc9-0933-40e0-958e-bc31edd1f3bf-dshm\") pod \"auth-test-model-f4502d-kserve-7944f5f479-zfv22\" (UID: \"15013cc9-0933-40e0-958e-bc31edd1f3bf\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:36:55.021687 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:55.021669    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15013cc9-0933-40e0-958e-bc31edd1f3bf-tls-certs\") pod \"auth-test-model-f4502d-kserve-7944f5f479-zfv22\" (UID: \"15013cc9-0933-40e0-958e-bc31edd1f3bf\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:36:55.026241 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:55.026218    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klhc2\" (UniqueName: \"kubernetes.io/projected/15013cc9-0933-40e0-958e-bc31edd1f3bf-kube-api-access-klhc2\") pod \"auth-test-model-f4502d-kserve-7944f5f479-zfv22\" (UID: \"15013cc9-0933-40e0-958e-bc31edd1f3bf\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:36:55.105735 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:55.105653    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:36:55.374649 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:55.374625    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"]
Sep 11 12:36:55.376918 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:36:55.376891    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15013cc9_0933_40e0_958e_bc31edd1f3bf.slice/crio-7f67139942fbe623dbb06ef42abab0fcbfabb0ea7c6963738694561fe11b9d33 WatchSource:0}: Error finding container 7f67139942fbe623dbb06ef42abab0fcbfabb0ea7c6963738694561fe11b9d33: Status 404 returned error can't find the container with id 7f67139942fbe623dbb06ef42abab0fcbfabb0ea7c6963738694561fe11b9d33
Sep 11 12:36:56.094638 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:56.094599    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22" event={"ID":"15013cc9-0933-40e0-958e-bc31edd1f3bf","Type":"ContainerStarted","Data":"f650ee7f12e6961e993b27e3d8892c030171e160a6a1c9c03ddde810a1604417"}
Sep 11 12:36:56.094999 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:56.094646    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22" event={"ID":"15013cc9-0933-40e0-958e-bc31edd1f3bf","Type":"ContainerStarted","Data":"7f67139942fbe623dbb06ef42abab0fcbfabb0ea7c6963738694561fe11b9d33"}
Sep 11 12:36:56.112847 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:36:56.112795    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22" podStartSLOduration=2.112778764 podStartE2EDuration="2.112778764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:36:56.110866304 +0000 UTC m=+1347.673646067" watchObservedRunningTime="2026-09-11 12:36:56.112778764 +0000 UTC m=+1347.675558526"
Sep 11 12:37:00.033253 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:00.033219    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:37:05.106448 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:05.106406    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:37:05.106448 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:05.106457    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:37:05.119046 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:05.119015    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:37:05.148136 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:05.148111    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:37:08.197611 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:08.197575    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"]
Sep 11 12:37:08.201036 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:08.201017    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:37:08.203688 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:08.203663    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cb20c8\"/\"distinct-2-w4-cb20c8-kserve-self-signed-certs\""
Sep 11 12:37:08.210974 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:08.210953    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"]
Sep 11 12:37:08.338081 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:08.338045    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92ad8452-ea85-436c-824c-e270e48a9c39-model-cache\") pod \"distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm\" (UID: \"92ad8452-ea85-436c-824c-e270e48a9c39\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:37:08.338293 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:08.338095    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92ad8452-ea85-436c-824c-e270e48a9c39-tmp-dir\") pod \"distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm\" (UID: \"92ad8452-ea85-436c-824c-e270e48a9c39\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:37:08.338293 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:08.338143    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92ad8452-ea85-436c-824c-e270e48a9c39-tls-certs\") pod \"distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm\" (UID: \"92ad8452-ea85-436c-824c-e270e48a9c39\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:37:08.338293 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:08.338248    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92ad8452-ea85-436c-824c-e270e48a9c39-home\") pod \"distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm\" (UID: \"92ad8452-ea85-436c-824c-e270e48a9c39\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:37:08.338293 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:08.338271    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvjk5\" (UniqueName: \"kubernetes.io/projected/92ad8452-ea85-436c-824c-e270e48a9c39-kube-api-access-xvjk5\") pod \"distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm\" (UID: \"92ad8452-ea85-436c-824c-e270e48a9c39\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:37:08.338463 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:08.338311    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92ad8452-ea85-436c-824c-e270e48a9c39-dshm\") pod \"distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm\" (UID: \"92ad8452-ea85-436c-824c-e270e48a9c39\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:37:08.439715 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:08.439680    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92ad8452-ea85-436c-824c-e270e48a9c39-home\") pod \"distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm\" (UID: \"92ad8452-ea85-436c-824c-e270e48a9c39\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:37:08.439901 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:08.439773    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvjk5\" (UniqueName: \"kubernetes.io/projected/92ad8452-ea85-436c-824c-e270e48a9c39-kube-api-access-xvjk5\") pod \"distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm\" (UID: \"92ad8452-ea85-436c-824c-e270e48a9c39\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:37:08.439901 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:08.439816    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92ad8452-ea85-436c-824c-e270e48a9c39-dshm\") pod \"distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm\" (UID: \"92ad8452-ea85-436c-824c-e270e48a9c39\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:37:08.439901 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:08.439893    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92ad8452-ea85-436c-824c-e270e48a9c39-model-cache\") pod \"distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm\" (UID: \"92ad8452-ea85-436c-824c-e270e48a9c39\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:37:08.440063 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:08.439920    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92ad8452-ea85-436c-824c-e270e48a9c39-tmp-dir\") pod \"distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm\" (UID: \"92ad8452-ea85-436c-824c-e270e48a9c39\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:37:08.440063 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:08.439957    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92ad8452-ea85-436c-824c-e270e48a9c39-tls-certs\") pod \"distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm\" (UID: \"92ad8452-ea85-436c-824c-e270e48a9c39\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:37:08.440165 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:08.440068    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92ad8452-ea85-436c-824c-e270e48a9c39-home\") pod \"distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm\" (UID: \"92ad8452-ea85-436c-824c-e270e48a9c39\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:37:08.440299 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:08.440275    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92ad8452-ea85-436c-824c-e270e48a9c39-model-cache\") pod \"distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm\" (UID: \"92ad8452-ea85-436c-824c-e270e48a9c39\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:37:08.440372 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:08.440296    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92ad8452-ea85-436c-824c-e270e48a9c39-tmp-dir\") pod \"distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm\" (UID: \"92ad8452-ea85-436c-824c-e270e48a9c39\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:37:08.442171 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:08.442142    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92ad8452-ea85-436c-824c-e270e48a9c39-dshm\") pod \"distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm\" (UID: \"92ad8452-ea85-436c-824c-e270e48a9c39\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:37:08.442408 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:08.442391    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92ad8452-ea85-436c-824c-e270e48a9c39-tls-certs\") pod \"distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm\" (UID: \"92ad8452-ea85-436c-824c-e270e48a9c39\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:37:08.447834 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:08.447784    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvjk5\" (UniqueName: \"kubernetes.io/projected/92ad8452-ea85-436c-824c-e270e48a9c39-kube-api-access-xvjk5\") pod \"distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm\" (UID: \"92ad8452-ea85-436c-824c-e270e48a9c39\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:37:08.511644 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:08.511624    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:37:08.782792 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:08.782767    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"]
Sep 11 12:37:08.785481 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:37:08.785452    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92ad8452_ea85_436c_824c_e270e48a9c39.slice/crio-6f3d1c158b9ba4abd494029a63c8de5ea4f80cd9505bb1926b15ca9101f8d966 WatchSource:0}: Error finding container 6f3d1c158b9ba4abd494029a63c8de5ea4f80cd9505bb1926b15ca9101f8d966: Status 404 returned error can't find the container with id 6f3d1c158b9ba4abd494029a63c8de5ea4f80cd9505bb1926b15ca9101f8d966
Sep 11 12:37:09.157328 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:09.157238    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm" event={"ID":"92ad8452-ea85-436c-824c-e270e48a9c39","Type":"ContainerStarted","Data":"772daab48494e55059cfc087aa2b56e480442f9c762313c9b95d10d8e87257ce"}
Sep 11 12:37:09.157328 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:09.157284    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm" event={"ID":"92ad8452-ea85-436c-824c-e270e48a9c39","Type":"ContainerStarted","Data":"6f3d1c158b9ba4abd494029a63c8de5ea4f80cd9505bb1926b15ca9101f8d966"}
Sep 11 12:37:09.173049 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:09.172989    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm" podStartSLOduration=1.172956752 podStartE2EDuration="1.172956752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:37:09.172915766 +0000 UTC m=+1360.735695529" watchObservedRunningTime="2026-09-11 12:37:09.172956752 +0000 UTC m=+1360.735736509"
Sep 11 12:37:18.512127 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:18.512091    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:37:18.512127 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:18.512140    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:37:18.524859 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:18.524835    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:37:19.216399 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:19.216371    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:37:39.326509 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:37:39.326477    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:38:10.747694 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:10.747651    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:38:17.921374 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:17.921338    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"]
Sep 11 12:38:17.924951 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:17.924930    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"
Sep 11 12:38:17.927480 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:17.927453    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 12:38:17.927598 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:17.927530    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f55824-serving-cert\""
Sep 11 12:38:17.927810 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:17.927791    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 12:38:17.927810 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:17.927802    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l5tx4\""
Sep 11 12:38:17.927955 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:17.927838    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 12:38:17.938182 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:17.938158    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"]
Sep 11 12:38:17.998050 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:17.998016    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381e25f8-6416-4d5d-9e14-58c31854ab47-maas-api-tls\") pod \"maas-api-e2e-api-a-f55824-56cf678746-lspqr\" (UID: \"381e25f8-6416-4d5d-9e14-58c31854ab47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"
Sep 11 12:38:17.998230 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:17.998062    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79knv\" (UniqueName: \"kubernetes.io/projected/381e25f8-6416-4d5d-9e14-58c31854ab47-kube-api-access-79knv\") pod \"maas-api-e2e-api-a-f55824-56cf678746-lspqr\" (UID: \"381e25f8-6416-4d5d-9e14-58c31854ab47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"
Sep 11 12:38:17.998230 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:17.998104    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381e25f8-6416-4d5d-9e14-58c31854ab47-openshift-service-ca\") pod \"maas-api-e2e-api-a-f55824-56cf678746-lspqr\" (UID: \"381e25f8-6416-4d5d-9e14-58c31854ab47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"
Sep 11 12:38:17.998230 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:17.998175    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/381e25f8-6416-4d5d-9e14-58c31854ab47-metrics-certs\") pod \"maas-api-e2e-api-a-f55824-56cf678746-lspqr\" (UID: \"381e25f8-6416-4d5d-9e14-58c31854ab47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"
Sep 11 12:38:18.098923 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.098892    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381e25f8-6416-4d5d-9e14-58c31854ab47-maas-api-tls\") pod \"maas-api-e2e-api-a-f55824-56cf678746-lspqr\" (UID: \"381e25f8-6416-4d5d-9e14-58c31854ab47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"
Sep 11 12:38:18.099043 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.098931    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79knv\" (UniqueName: \"kubernetes.io/projected/381e25f8-6416-4d5d-9e14-58c31854ab47-kube-api-access-79knv\") pod \"maas-api-e2e-api-a-f55824-56cf678746-lspqr\" (UID: \"381e25f8-6416-4d5d-9e14-58c31854ab47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"
Sep 11 12:38:18.099043 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.098958    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381e25f8-6416-4d5d-9e14-58c31854ab47-openshift-service-ca\") pod \"maas-api-e2e-api-a-f55824-56cf678746-lspqr\" (UID: \"381e25f8-6416-4d5d-9e14-58c31854ab47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"
Sep 11 12:38:18.099043 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:38:18.099025    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-serving-cert: secret "maas-api-e2e-api-a-f55824-serving-cert" not found
Sep 11 12:38:18.099164 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:38:18.099090    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/381e25f8-6416-4d5d-9e14-58c31854ab47-maas-api-tls podName:381e25f8-6416-4d5d-9e14-58c31854ab47 nodeName:}" failed. No retries permitted until 2026-09-11 12:38:18.599073378 +0000 UTC m=+1430.161853156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/381e25f8-6416-4d5d-9e14-58c31854ab47-maas-api-tls") pod "maas-api-e2e-api-a-f55824-56cf678746-lspqr" (UID: "381e25f8-6416-4d5d-9e14-58c31854ab47") : secret "maas-api-e2e-api-a-f55824-serving-cert" not found
Sep 11 12:38:18.099164 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.099131    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/381e25f8-6416-4d5d-9e14-58c31854ab47-metrics-certs\") pod \"maas-api-e2e-api-a-f55824-56cf678746-lspqr\" (UID: \"381e25f8-6416-4d5d-9e14-58c31854ab47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"
Sep 11 12:38:18.099543 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.099525    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381e25f8-6416-4d5d-9e14-58c31854ab47-openshift-service-ca\") pod \"maas-api-e2e-api-a-f55824-56cf678746-lspqr\" (UID: \"381e25f8-6416-4d5d-9e14-58c31854ab47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"
Sep 11 12:38:18.101639 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.101617    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/381e25f8-6416-4d5d-9e14-58c31854ab47-metrics-certs\") pod \"maas-api-e2e-api-a-f55824-56cf678746-lspqr\" (UID: \"381e25f8-6416-4d5d-9e14-58c31854ab47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"
Sep 11 12:38:18.109540 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.109517    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79knv\" (UniqueName: \"kubernetes.io/projected/381e25f8-6416-4d5d-9e14-58c31854ab47-kube-api-access-79knv\") pod \"maas-api-e2e-api-a-f55824-56cf678746-lspqr\" (UID: \"381e25f8-6416-4d5d-9e14-58c31854ab47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"
Sep 11 12:38:18.548051 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.548023    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b"]
Sep 11 12:38:18.551549 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.551533    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b"
Sep 11 12:38:18.553833 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.553812    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f55824-dockercfg-7txbk\""
Sep 11 12:38:18.554176 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.554161    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f55824\""
Sep 11 12:38:18.562676 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.562656    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b"]
Sep 11 12:38:18.579964 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.579945    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f55824-749d4fc899-c48lh"]
Sep 11 12:38:18.583121 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.583108    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f55824-749d4fc899-c48lh"
Sep 11 12:38:18.599121 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.599101    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f55824-749d4fc899-c48lh"]
Sep 11 12:38:18.603298 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.603229    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbnmn\" (UniqueName: \"kubernetes.io/projected/25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2-kube-api-access-wbnmn\") pod \"payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b\" (UID: \"25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b"
Sep 11 12:38:18.603410 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.603318    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381e25f8-6416-4d5d-9e14-58c31854ab47-maas-api-tls\") pod \"maas-api-e2e-api-a-f55824-56cf678746-lspqr\" (UID: \"381e25f8-6416-4d5d-9e14-58c31854ab47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"
Sep 11 12:38:18.603410 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.603354    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea6c184-9fb1-4227-ae7e-edd9aede0d7f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f55824-749d4fc899-c48lh\" (UID: \"cea6c184-9fb1-4227-ae7e-edd9aede0d7f\") " pod="openshift-ingress/payload-processing-e2e-api-a-f55824-749d4fc899-c48lh"
Sep 11 12:38:18.603410 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.603383    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b\" (UID: \"25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b"
Sep 11 12:38:18.603587 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.603449    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5jhm\" (UniqueName: \"kubernetes.io/projected/cea6c184-9fb1-4227-ae7e-edd9aede0d7f-kube-api-access-m5jhm\") pod \"payload-processing-e2e-api-a-f55824-749d4fc899-c48lh\" (UID: \"cea6c184-9fb1-4227-ae7e-edd9aede0d7f\") " pod="openshift-ingress/payload-processing-e2e-api-a-f55824-749d4fc899-c48lh"
Sep 11 12:38:18.605763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.605740    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381e25f8-6416-4d5d-9e14-58c31854ab47-maas-api-tls\") pod \"maas-api-e2e-api-a-f55824-56cf678746-lspqr\" (UID: \"381e25f8-6416-4d5d-9e14-58c31854ab47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"
Sep 11 12:38:18.704361 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.704331    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea6c184-9fb1-4227-ae7e-edd9aede0d7f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f55824-749d4fc899-c48lh\" (UID: \"cea6c184-9fb1-4227-ae7e-edd9aede0d7f\") " pod="openshift-ingress/payload-processing-e2e-api-a-f55824-749d4fc899-c48lh"
Sep 11 12:38:18.704479 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.704363    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b\" (UID: \"25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b"
Sep 11 12:38:18.704479 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.704397    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5jhm\" (UniqueName: \"kubernetes.io/projected/cea6c184-9fb1-4227-ae7e-edd9aede0d7f-kube-api-access-m5jhm\") pod \"payload-processing-e2e-api-a-f55824-749d4fc899-c48lh\" (UID: \"cea6c184-9fb1-4227-ae7e-edd9aede0d7f\") " pod="openshift-ingress/payload-processing-e2e-api-a-f55824-749d4fc899-c48lh"
Sep 11 12:38:18.704479 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.704450    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbnmn\" (UniqueName: \"kubernetes.io/projected/25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2-kube-api-access-wbnmn\") pod \"payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b\" (UID: \"25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b"
Sep 11 12:38:18.705061 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.705039    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b\" (UID: \"25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b"
Sep 11 12:38:18.705118 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.705045    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea6c184-9fb1-4227-ae7e-edd9aede0d7f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f55824-749d4fc899-c48lh\" (UID: \"cea6c184-9fb1-4227-ae7e-edd9aede0d7f\") " pod="openshift-ingress/payload-processing-e2e-api-a-f55824-749d4fc899-c48lh"
Sep 11 12:38:18.712174 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.712155    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5jhm\" (UniqueName: \"kubernetes.io/projected/cea6c184-9fb1-4227-ae7e-edd9aede0d7f-kube-api-access-m5jhm\") pod \"payload-processing-e2e-api-a-f55824-749d4fc899-c48lh\" (UID: \"cea6c184-9fb1-4227-ae7e-edd9aede0d7f\") " pod="openshift-ingress/payload-processing-e2e-api-a-f55824-749d4fc899-c48lh"
Sep 11 12:38:18.712259 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.712228    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbnmn\" (UniqueName: \"kubernetes.io/projected/25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2-kube-api-access-wbnmn\") pod \"payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b\" (UID: \"25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b"
Sep 11 12:38:18.836440 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.836359    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"
Sep 11 12:38:18.862401 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.862370    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b"
Sep 11 12:38:18.892387 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:18.892358    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f55824-749d4fc899-c48lh"
Sep 11 12:38:19.114859 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:19.114724    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"]
Sep 11 12:38:19.117331 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:38:19.117305    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381e25f8_6416_4d5d_9e14_58c31854ab47.slice/crio-9bf24666b509fe439989eba93f18e41500222c268229f5f32981281684b18717 WatchSource:0}: Error finding container 9bf24666b509fe439989eba93f18e41500222c268229f5f32981281684b18717: Status 404 returned error can't find the container with id 9bf24666b509fe439989eba93f18e41500222c268229f5f32981281684b18717
Sep 11 12:38:19.146224 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:19.146202    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b"]
Sep 11 12:38:19.148256 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:38:19.148232    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c1cfad_6e48_4fb9_bbce_5c19c0c0b6a2.slice/crio-cf5c334324037208c52d826daa93b784c95a099e0478fdca8e1b51a53690c25a WatchSource:0}: Error finding container cf5c334324037208c52d826daa93b784c95a099e0478fdca8e1b51a53690c25a: Status 404 returned error can't find the container with id cf5c334324037208c52d826daa93b784c95a099e0478fdca8e1b51a53690c25a
Sep 11 12:38:19.182765 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:19.182742    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f55824-749d4fc899-c48lh"]
Sep 11 12:38:19.186412 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:38:19.186388    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea6c184_9fb1_4227_ae7e_edd9aede0d7f.slice/crio-3da863289a9e86f5eed79cab91dfc9f53dbb4a167460dc75870c1a4a82af74c8 WatchSource:0}: Error finding container 3da863289a9e86f5eed79cab91dfc9f53dbb4a167460dc75870c1a4a82af74c8: Status 404 returned error can't find the container with id 3da863289a9e86f5eed79cab91dfc9f53dbb4a167460dc75870c1a4a82af74c8
Sep 11 12:38:19.516353 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:19.516314    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b" event={"ID":"25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2","Type":"ContainerStarted","Data":"76f81870ac9cee4f5571021c9e7a3a9547e0c58558992a8d2ecf578e079c78cc"}
Sep 11 12:38:19.516545 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:19.516380    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b" event={"ID":"25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2","Type":"ContainerStarted","Data":"cf5c334324037208c52d826daa93b784c95a099e0478fdca8e1b51a53690c25a"}
Sep 11 12:38:19.516745 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:19.516722    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b"
Sep 11 12:38:19.518728 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:19.518670    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f55824-749d4fc899-c48lh" event={"ID":"cea6c184-9fb1-4227-ae7e-edd9aede0d7f","Type":"ContainerStarted","Data":"45343ad9734ff02e013e4f861a65d18da534fea457b6424731dbc4257d2e1a85"}
Sep 11 12:38:19.518728 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:19.518726    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f55824-749d4fc899-c48lh" event={"ID":"cea6c184-9fb1-4227-ae7e-edd9aede0d7f","Type":"ContainerStarted","Data":"3da863289a9e86f5eed79cab91dfc9f53dbb4a167460dc75870c1a4a82af74c8"}
Sep 11 12:38:19.518927 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:19.518754    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f55824-749d4fc899-c48lh"
Sep 11 12:38:19.520289 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:19.520266    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr" event={"ID":"381e25f8-6416-4d5d-9e14-58c31854ab47","Type":"ContainerStarted","Data":"9bf24666b509fe439989eba93f18e41500222c268229f5f32981281684b18717"}
Sep 11 12:38:19.541959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:19.541913    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b" podStartSLOduration=1.541896596 podStartE2EDuration="1.541896596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:38:19.540451158 +0000 UTC m=+1431.103230920" watchObservedRunningTime="2026-09-11 12:38:19.541896596 +0000 UTC m=+1431.104676358"
Sep 11 12:38:19.560684 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:19.560629    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f55824-749d4fc899-c48lh" podStartSLOduration=1.5606156530000002 podStartE2EDuration="1.560615653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:38:19.559256831 +0000 UTC m=+1431.122036596" watchObservedRunningTime="2026-09-11 12:38:19.560615653 +0000 UTC m=+1431.123395415"
Sep 11 12:38:20.528526 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:20.528488    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr" event={"ID":"381e25f8-6416-4d5d-9e14-58c31854ab47","Type":"ContainerStarted","Data":"b6955156b8490d35cc63e136e1f4430669539d8acc06f844d4687fc0e3c8adf5"}
Sep 11 12:38:20.528971 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:20.528607    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"
Sep 11 12:38:20.549952 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:20.549901    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr" podStartSLOduration=3.121494661 podStartE2EDuration="3.54988658s" totalImagesPullingTime="428.391919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:38:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:38:20.547821297 +0000 UTC m=+1432.110601062" watchObservedRunningTime="2026-09-11 12:38:20.54988658 +0000 UTC m=+1432.112666342"
Sep 11 12:38:26.539560 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:26.539478    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"
Sep 11 12:38:30.530261 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:30.530229    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f55824-749d4fc899-c48lh"
Sep 11 12:38:30.530651 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:30.530303    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b"
Sep 11 12:38:47.612342 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:47.612311    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp"]
Sep 11 12:38:47.615958 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:47.615940    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp"
Sep 11 12:38:47.618424 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:47.618403    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d18c79\""
Sep 11 12:38:47.629566 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:47.629544    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp"]
Sep 11 12:38:47.782463 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:47.782423    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r2c8\" (UniqueName: \"kubernetes.io/projected/1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca-kube-api-access-5r2c8\") pod \"payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp\" (UID: \"1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp"
Sep 11 12:38:47.782620 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:47.782482    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp\" (UID: \"1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp"
Sep 11 12:38:47.883721 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:47.883644    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r2c8\" (UniqueName: \"kubernetes.io/projected/1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca-kube-api-access-5r2c8\") pod \"payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp\" (UID: \"1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp"
Sep 11 12:38:47.883721 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:47.883681    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp\" (UID: \"1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp"
Sep 11 12:38:47.884356 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:47.884330    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp\" (UID: \"1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp"
Sep 11 12:38:47.891670 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:47.891647    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r2c8\" (UniqueName: \"kubernetes.io/projected/1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca-kube-api-access-5r2c8\") pod \"payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp\" (UID: \"1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp"
Sep 11 12:38:47.926492 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:47.926467    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp"
Sep 11 12:38:48.199149 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:48.199125    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp"]
Sep 11 12:38:48.201579 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:38:48.201551    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ffdd095_e1c7_4ec2_98ce_d0832c5ca0ca.slice/crio-499512f8e51f837aaa9594132da68c7ab5afe4e57ca0b11b85cf64265cd1f4cd WatchSource:0}: Error finding container 499512f8e51f837aaa9594132da68c7ab5afe4e57ca0b11b85cf64265cd1f4cd: Status 404 returned error can't find the container with id 499512f8e51f837aaa9594132da68c7ab5afe4e57ca0b11b85cf64265cd1f4cd
Sep 11 12:38:48.677125 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:48.677082    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp" event={"ID":"1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca","Type":"ContainerStarted","Data":"eb90215bc74182be9381f2b827a75aaeed7cb90fb93a740b3e81bcb44edc9cef"}
Sep 11 12:38:48.677621 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:48.677143    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp" event={"ID":"1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca","Type":"ContainerStarted","Data":"499512f8e51f837aaa9594132da68c7ab5afe4e57ca0b11b85cf64265cd1f4cd"}
Sep 11 12:38:48.677621 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:48.677170    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp"
Sep 11 12:38:48.696153 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:48.696093    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp" podStartSLOduration=1.6960727740000001 podStartE2EDuration="1.696072774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:38:48.695490382 +0000 UTC m=+1460.258270148" watchObservedRunningTime="2026-09-11 12:38:48.696072774 +0000 UTC m=+1460.258852539"
Sep 11 12:38:59.685362 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:38:59.685320    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp"
Sep 11 12:39:26.840741 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:26.840708    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:39:28.956357 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:28.956330    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcfm6_88402687-4e5b-411c-be7a-4cb955f90425/ovn-acl-logging/0.log"
Sep 11 12:39:28.960772 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:28.960749    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcfm6_88402687-4e5b-411c-be7a-4cb955f90425/ovn-acl-logging/0.log"
Sep 11 12:39:29.650333 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:29.650301    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"]
Sep 11 12:39:29.650546 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:29.650505    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr" podUID="381e25f8-6416-4d5d-9e14-58c31854ab47" containerName="maas-api" containerID="cri-o://b6955156b8490d35cc63e136e1f4430669539d8acc06f844d4687fc0e3c8adf5" gracePeriod=30
Sep 11 12:39:29.701502 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:29.701474    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f55824-749d4fc899-c48lh"]
Sep 11 12:39:29.701697 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:29.701674    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f55824-749d4fc899-c48lh" podUID="cea6c184-9fb1-4227-ae7e-edd9aede0d7f" containerName="payload-processing" containerID="cri-o://45343ad9734ff02e013e4f861a65d18da534fea457b6424731dbc4257d2e1a85" gracePeriod=30
Sep 11 12:39:29.706171 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:29.706147    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b"]
Sep 11 12:39:29.706428 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:29.706379    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b" podUID="25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2" containerName="payload-pre-processing" containerID="cri-o://76f81870ac9cee4f5571021c9e7a3a9547e0c58558992a8d2ecf578e079c78cc" gracePeriod=30
Sep 11 12:39:29.892445 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:29.892416    2561 generic.go:356] "Generic (PLEG): container finished" podID="25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2" containerID="76f81870ac9cee4f5571021c9e7a3a9547e0c58558992a8d2ecf578e079c78cc" exitCode=0
Sep 11 12:39:29.892591 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:29.892483    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b" event={"ID":"25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2","Type":"ContainerDied","Data":"76f81870ac9cee4f5571021c9e7a3a9547e0c58558992a8d2ecf578e079c78cc"}
Sep 11 12:39:29.894052 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:29.894029    2561 generic.go:356] "Generic (PLEG): container finished" podID="cea6c184-9fb1-4227-ae7e-edd9aede0d7f" containerID="45343ad9734ff02e013e4f861a65d18da534fea457b6424731dbc4257d2e1a85" exitCode=0
Sep 11 12:39:29.894167 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:29.894093    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f55824-749d4fc899-c48lh" event={"ID":"cea6c184-9fb1-4227-ae7e-edd9aede0d7f","Type":"ContainerDied","Data":"45343ad9734ff02e013e4f861a65d18da534fea457b6424731dbc4257d2e1a85"}
Sep 11 12:39:29.895621 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:29.895600    2561 generic.go:356] "Generic (PLEG): container finished" podID="381e25f8-6416-4d5d-9e14-58c31854ab47" containerID="b6955156b8490d35cc63e136e1f4430669539d8acc06f844d4687fc0e3c8adf5" exitCode=0
Sep 11 12:39:29.895718 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:29.895655    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr" event={"ID":"381e25f8-6416-4d5d-9e14-58c31854ab47","Type":"ContainerDied","Data":"b6955156b8490d35cc63e136e1f4430669539d8acc06f844d4687fc0e3c8adf5"}
Sep 11 12:39:29.987251 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:29.987221    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f55824-749d4fc899-c48lh"
Sep 11 12:39:29.990885 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:29.990864    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b"
Sep 11 12:39:30.011611 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.011587    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"
Sep 11 12:39:30.072137 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.072108    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5jhm\" (UniqueName: \"kubernetes.io/projected/cea6c184-9fb1-4227-ae7e-edd9aede0d7f-kube-api-access-m5jhm\") pod \"cea6c184-9fb1-4227-ae7e-edd9aede0d7f\" (UID: \"cea6c184-9fb1-4227-ae7e-edd9aede0d7f\") "
Sep 11 12:39:30.072312 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.072211    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2-plugins-config-volume\") pod \"25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2\" (UID: \"25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2\") "
Sep 11 12:39:30.072312 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.072241    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea6c184-9fb1-4227-ae7e-edd9aede0d7f-plugins-config-volume\") pod \"cea6c184-9fb1-4227-ae7e-edd9aede0d7f\" (UID: \"cea6c184-9fb1-4227-ae7e-edd9aede0d7f\") "
Sep 11 12:39:30.072312 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.072265    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbnmn\" (UniqueName: \"kubernetes.io/projected/25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2-kube-api-access-wbnmn\") pod \"25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2\" (UID: \"25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2\") "
Sep 11 12:39:30.072640 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.072616    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cea6c184-9fb1-4227-ae7e-edd9aede0d7f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cea6c184-9fb1-4227-ae7e-edd9aede0d7f" (UID: "cea6c184-9fb1-4227-ae7e-edd9aede0d7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:39:30.072640 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.072627    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2" (UID: "25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:39:30.074212 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.074157    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea6c184-9fb1-4227-ae7e-edd9aede0d7f-kube-api-access-m5jhm" (OuterVolumeSpecName: "kube-api-access-m5jhm") pod "cea6c184-9fb1-4227-ae7e-edd9aede0d7f" (UID: "cea6c184-9fb1-4227-ae7e-edd9aede0d7f"). InnerVolumeSpecName "kube-api-access-m5jhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:39:30.074315 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.074286    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2-kube-api-access-wbnmn" (OuterVolumeSpecName: "kube-api-access-wbnmn") pod "25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2" (UID: "25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2"). InnerVolumeSpecName "kube-api-access-wbnmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:39:30.173323 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.173247    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381e25f8-6416-4d5d-9e14-58c31854ab47-openshift-service-ca\") pod \"381e25f8-6416-4d5d-9e14-58c31854ab47\" (UID: \"381e25f8-6416-4d5d-9e14-58c31854ab47\") "
Sep 11 12:39:30.173451 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.173336    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381e25f8-6416-4d5d-9e14-58c31854ab47-maas-api-tls\") pod \"381e25f8-6416-4d5d-9e14-58c31854ab47\" (UID: \"381e25f8-6416-4d5d-9e14-58c31854ab47\") "
Sep 11 12:39:30.173451 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.173444    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79knv\" (UniqueName: \"kubernetes.io/projected/381e25f8-6416-4d5d-9e14-58c31854ab47-kube-api-access-79knv\") pod \"381e25f8-6416-4d5d-9e14-58c31854ab47\" (UID: \"381e25f8-6416-4d5d-9e14-58c31854ab47\") "
Sep 11 12:39:30.173527 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.173476    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/381e25f8-6416-4d5d-9e14-58c31854ab47-metrics-certs\") pod \"381e25f8-6416-4d5d-9e14-58c31854ab47\" (UID: \"381e25f8-6416-4d5d-9e14-58c31854ab47\") "
Sep 11 12:39:30.173662 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.173640    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381e25f8-6416-4d5d-9e14-58c31854ab47-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "381e25f8-6416-4d5d-9e14-58c31854ab47" (UID: "381e25f8-6416-4d5d-9e14-58c31854ab47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:39:30.173864 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.173847    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5jhm\" (UniqueName: \"kubernetes.io/projected/cea6c184-9fb1-4227-ae7e-edd9aede0d7f-kube-api-access-m5jhm\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:30.173927 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.173866    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381e25f8-6416-4d5d-9e14-58c31854ab47-openshift-service-ca\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:30.173927 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.173875    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:30.173927 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.173884    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea6c184-9fb1-4227-ae7e-edd9aede0d7f-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:30.173927 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.173893    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbnmn\" (UniqueName: \"kubernetes.io/projected/25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2-kube-api-access-wbnmn\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:30.175460 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.175434    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/381e25f8-6416-4d5d-9e14-58c31854ab47-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "381e25f8-6416-4d5d-9e14-58c31854ab47" (UID: "381e25f8-6416-4d5d-9e14-58c31854ab47"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:39:30.175566 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.175479    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/381e25f8-6416-4d5d-9e14-58c31854ab47-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "381e25f8-6416-4d5d-9e14-58c31854ab47" (UID: "381e25f8-6416-4d5d-9e14-58c31854ab47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:39:30.175566 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.175483    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381e25f8-6416-4d5d-9e14-58c31854ab47-kube-api-access-79knv" (OuterVolumeSpecName: "kube-api-access-79knv") pod "381e25f8-6416-4d5d-9e14-58c31854ab47" (UID: "381e25f8-6416-4d5d-9e14-58c31854ab47"). InnerVolumeSpecName "kube-api-access-79knv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:39:30.274618 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.274584    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381e25f8-6416-4d5d-9e14-58c31854ab47-maas-api-tls\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:30.274618 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.274605    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79knv\" (UniqueName: \"kubernetes.io/projected/381e25f8-6416-4d5d-9e14-58c31854ab47-kube-api-access-79knv\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:30.274618 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.274615    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/381e25f8-6416-4d5d-9e14-58c31854ab47-metrics-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:30.902906 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.902877    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f55824-749d4fc899-c48lh"
Sep 11 12:39:30.903119 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.902882    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f55824-749d4fc899-c48lh" event={"ID":"cea6c184-9fb1-4227-ae7e-edd9aede0d7f","Type":"ContainerDied","Data":"3da863289a9e86f5eed79cab91dfc9f53dbb4a167460dc75870c1a4a82af74c8"}
Sep 11 12:39:30.903119 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.903019    2561 scope.go:117] "RemoveContainer" containerID="45343ad9734ff02e013e4f861a65d18da534fea457b6424731dbc4257d2e1a85"
Sep 11 12:39:30.904498 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.904452    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"
Sep 11 12:39:30.904568 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.904496    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr" event={"ID":"381e25f8-6416-4d5d-9e14-58c31854ab47","Type":"ContainerDied","Data":"9bf24666b509fe439989eba93f18e41500222c268229f5f32981281684b18717"}
Sep 11 12:39:30.906410 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.906376    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b"
Sep 11 12:39:30.906658 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.906374    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b" event={"ID":"25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2","Type":"ContainerDied","Data":"cf5c334324037208c52d826daa93b784c95a099e0478fdca8e1b51a53690c25a"}
Sep 11 12:39:30.914839 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.914815    2561 scope.go:117] "RemoveContainer" containerID="b6955156b8490d35cc63e136e1f4430669539d8acc06f844d4687fc0e3c8adf5"
Sep 11 12:39:30.923856 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.923842    2561 scope.go:117] "RemoveContainer" containerID="76f81870ac9cee4f5571021c9e7a3a9547e0c58558992a8d2ecf578e079c78cc"
Sep 11 12:39:30.933231 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.933213    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f55824-749d4fc899-c48lh"]
Sep 11 12:39:30.933845 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.933825    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f55824-749d4fc899-c48lh"]
Sep 11 12:39:30.944046 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.944026    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"]
Sep 11 12:39:30.954610 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.954590    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f55824-56cf678746-lspqr"]
Sep 11 12:39:30.964862 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.964840    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b"]
Sep 11 12:39:30.968322 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:30.968303    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f55824-5ddd979db6-bv59b"]
Sep 11 12:39:32.932152 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:32.932119    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2" path="/var/lib/kubelet/pods/25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2/volumes"
Sep 11 12:39:32.932573 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:32.932559    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="381e25f8-6416-4d5d-9e14-58c31854ab47" path="/var/lib/kubelet/pods/381e25f8-6416-4d5d-9e14-58c31854ab47/volumes"
Sep 11 12:39:32.932886 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:32.932874    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cea6c184-9fb1-4227-ae7e-edd9aede0d7f" path="/var/lib/kubelet/pods/cea6c184-9fb1-4227-ae7e-edd9aede0d7f/volumes"
Sep 11 12:39:34.290114 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:34.290078    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1a8ee3-bk4kg"]
Sep 11 12:39:34.290496 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:34.290479    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cea6c184-9fb1-4227-ae7e-edd9aede0d7f" containerName="payload-processing"
Sep 11 12:39:34.290496 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:34.290491    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cea6c184-9fb1-4227-ae7e-edd9aede0d7f" containerName="payload-processing"
Sep 11 12:39:34.290582 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:34.290499    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="381e25f8-6416-4d5d-9e14-58c31854ab47" containerName="maas-api"
Sep 11 12:39:34.290582 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:34.290506    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="381e25f8-6416-4d5d-9e14-58c31854ab47" containerName="maas-api"
Sep 11 12:39:34.290582 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:34.290513    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2" containerName="payload-pre-processing"
Sep 11 12:39:34.290582 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:34.290520    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2" containerName="payload-pre-processing"
Sep 11 12:39:34.290707 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:34.290588    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="381e25f8-6416-4d5d-9e14-58c31854ab47" containerName="maas-api"
Sep 11 12:39:34.290707 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:34.290599    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cea6c184-9fb1-4227-ae7e-edd9aede0d7f" containerName="payload-processing"
Sep 11 12:39:34.290707 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:34.290608    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="25c1cfad-6e48-4fb9-bbce-5c19c0c0b6a2" containerName="payload-pre-processing"
Sep 11 12:39:34.295244 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:34.295228    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1a8ee3-bk4kg"
Sep 11 12:39:34.298693 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:34.298410    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 12:39:34.298693 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:34.298439    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\""
Sep 11 12:39:34.299983 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:34.299958    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1a8ee3-bk4kg"]
Sep 11 12:39:34.414470 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:34.414435    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e102d7-70a9-46ac-af43-b3ef97ef6c74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1a8ee3-bk4kg\" (UID: \"f8e102d7-70a9-46ac-af43-b3ef97ef6c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1a8ee3-bk4kg"
Sep 11 12:39:34.515052 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:34.515023    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e102d7-70a9-46ac-af43-b3ef97ef6c74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1a8ee3-bk4kg\" (UID: \"f8e102d7-70a9-46ac-af43-b3ef97ef6c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1a8ee3-bk4kg"
Sep 11 12:39:34.515639 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:34.515619    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e102d7-70a9-46ac-af43-b3ef97ef6c74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1a8ee3-bk4kg\" (UID: \"f8e102d7-70a9-46ac-af43-b3ef97ef6c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1a8ee3-bk4kg"
Sep 11 12:39:34.607084 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:34.607012    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1a8ee3-bk4kg"
Sep 11 12:39:34.877953 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:34.877926    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1a8ee3-bk4kg"]
Sep 11 12:39:34.879968 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:39:34.879934    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e102d7_70a9_46ac_af43_b3ef97ef6c74.slice/crio-f23f982a74cde7fb0c0689b6875eb7d30034d156b3e51a378a0448e397a85a8e WatchSource:0}: Error finding container f23f982a74cde7fb0c0689b6875eb7d30034d156b3e51a378a0448e397a85a8e: Status 404 returned error can't find the container with id f23f982a74cde7fb0c0689b6875eb7d30034d156b3e51a378a0448e397a85a8e
Sep 11 12:39:34.932210 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:34.932164    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1a8ee3-bk4kg" event={"ID":"f8e102d7-70a9-46ac-af43-b3ef97ef6c74","Type":"ContainerStarted","Data":"f23f982a74cde7fb0c0689b6875eb7d30034d156b3e51a378a0448e397a85a8e"}
Sep 11 12:39:35.936968 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:35.936882    2561 generic.go:356] "Generic (PLEG): container finished" podID="f8e102d7-70a9-46ac-af43-b3ef97ef6c74" containerID="bc2d0531d2fdb1f3c69d861449255335d8bcbc90dff9e31d74f07b276b891142" exitCode=0
Sep 11 12:39:35.936968 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:35.936925    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1a8ee3-bk4kg" event={"ID":"f8e102d7-70a9-46ac-af43-b3ef97ef6c74","Type":"ContainerDied","Data":"bc2d0531d2fdb1f3c69d861449255335d8bcbc90dff9e31d74f07b276b891142"}
Sep 11 12:39:37.075629 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:37.075605    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1a8ee3-bk4kg"
Sep 11 12:39:37.139813 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:37.139783    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e102d7-70a9-46ac-af43-b3ef97ef6c74-maas-api-service-ca\") pod \"f8e102d7-70a9-46ac-af43-b3ef97ef6c74\" (UID: \"f8e102d7-70a9-46ac-af43-b3ef97ef6c74\") "
Sep 11 12:39:37.140168 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:37.140148    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e102d7-70a9-46ac-af43-b3ef97ef6c74-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8e102d7-70a9-46ac-af43-b3ef97ef6c74" (UID: "f8e102d7-70a9-46ac-af43-b3ef97ef6c74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:39:37.240961 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:37.240935    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e102d7-70a9-46ac-af43-b3ef97ef6c74-maas-api-service-ca\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:37.948608 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:37.948573    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1a8ee3-bk4kg" event={"ID":"f8e102d7-70a9-46ac-af43-b3ef97ef6c74","Type":"ContainerDied","Data":"f23f982a74cde7fb0c0689b6875eb7d30034d156b3e51a378a0448e397a85a8e"}
Sep 11 12:39:37.948822 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:37.948615    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1a8ee3-bk4kg"
Sep 11 12:39:37.948822 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:37.948619    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f23f982a74cde7fb0c0689b6875eb7d30034d156b3e51a378a0448e397a85a8e"
Sep 11 12:39:37.974753 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:39:37.974728    2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e102d7_70a9_46ac_af43_b3ef97ef6c74.slice\": RecentStats: unable to find data in memory cache]"
Sep 11 12:39:39.180843 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:39.180799    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"]
Sep 11 12:39:39.199527 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:39.199496    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"]
Sep 11 12:39:39.209553 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:39.209530    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"]
Sep 11 12:39:50.341267 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:50.341230    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1a8ee3-bk4kg"]
Sep 11 12:39:50.344654 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:50.344623    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1a8ee3-bk4kg"]
Sep 11 12:39:50.932175 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:50.932141    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8e102d7-70a9-46ac-af43-b3ef97ef6c74" path="/var/lib/kubelet/pods/f8e102d7-70a9-46ac-af43-b3ef97ef6c74/volumes"
Sep 11 12:39:52.675560 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:52.675520    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp"]
Sep 11 12:39:52.676077 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:52.675731    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp" podUID="1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca" containerName="payload-pre-processing" containerID="cri-o://eb90215bc74182be9381f2b827a75aaeed7cb90fb93a740b3e81bcb44edc9cef" gracePeriod=30
Sep 11 12:39:52.954575 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:52.954549    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp"
Sep 11 12:39:53.023543 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:53.023512    2561 generic.go:356] "Generic (PLEG): container finished" podID="1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca" containerID="eb90215bc74182be9381f2b827a75aaeed7cb90fb93a740b3e81bcb44edc9cef" exitCode=0
Sep 11 12:39:53.023543 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:53.023545    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp"
Sep 11 12:39:53.023741 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:53.023593    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp" event={"ID":"1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca","Type":"ContainerDied","Data":"eb90215bc74182be9381f2b827a75aaeed7cb90fb93a740b3e81bcb44edc9cef"}
Sep 11 12:39:53.023741 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:53.023634    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp" event={"ID":"1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca","Type":"ContainerDied","Data":"499512f8e51f837aaa9594132da68c7ab5afe4e57ca0b11b85cf64265cd1f4cd"}
Sep 11 12:39:53.023741 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:53.023657    2561 scope.go:117] "RemoveContainer" containerID="eb90215bc74182be9381f2b827a75aaeed7cb90fb93a740b3e81bcb44edc9cef"
Sep 11 12:39:53.032623 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:53.032601    2561 scope.go:117] "RemoveContainer" containerID="eb90215bc74182be9381f2b827a75aaeed7cb90fb93a740b3e81bcb44edc9cef"
Sep 11 12:39:53.032883 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:39:53.032862    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb90215bc74182be9381f2b827a75aaeed7cb90fb93a740b3e81bcb44edc9cef\": container with ID starting with eb90215bc74182be9381f2b827a75aaeed7cb90fb93a740b3e81bcb44edc9cef not found: ID does not exist" containerID="eb90215bc74182be9381f2b827a75aaeed7cb90fb93a740b3e81bcb44edc9cef"
Sep 11 12:39:53.032929 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:53.032897    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb90215bc74182be9381f2b827a75aaeed7cb90fb93a740b3e81bcb44edc9cef"} err="failed to get container status \"eb90215bc74182be9381f2b827a75aaeed7cb90fb93a740b3e81bcb44edc9cef\": rpc error: code = NotFound desc = could not find container \"eb90215bc74182be9381f2b827a75aaeed7cb90fb93a740b3e81bcb44edc9cef\": container with ID starting with eb90215bc74182be9381f2b827a75aaeed7cb90fb93a740b3e81bcb44edc9cef not found: ID does not exist"
Sep 11 12:39:53.078449 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:53.078429    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca-plugins-config-volume\") pod \"1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca\" (UID: \"1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca\") "
Sep 11 12:39:53.078580 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:53.078563    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r2c8\" (UniqueName: \"kubernetes.io/projected/1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca-kube-api-access-5r2c8\") pod \"1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca\" (UID: \"1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca\") "
Sep 11 12:39:53.078840 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:53.078810    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca" (UID: "1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:39:53.078935 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:53.078888    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:53.080505 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:53.080475    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca-kube-api-access-5r2c8" (OuterVolumeSpecName: "kube-api-access-5r2c8") pod "1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca" (UID: "1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca"). InnerVolumeSpecName "kube-api-access-5r2c8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:39:53.179627 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:53.179602    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r2c8\" (UniqueName: \"kubernetes.io/projected/1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca-kube-api-access-5r2c8\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:53.347759 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:53.347729    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp"]
Sep 11 12:39:53.351554 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:53.351521    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d18c79-67f9777794-wwdlp"]
Sep 11 12:39:54.200392 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.200357    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58" podUID="fb3646b9-acb4-43d6-8f30-768306fdb4b3" containerName="main" containerID="cri-o://42983d8339d2c5a8b63244882b1cfaf484c6aec5eff91849a9714f19a38edd25" gracePeriod=45
Sep 11 12:39:54.224579 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.224540    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj" podUID="0d5f9a4b-ce98-4594-ac66-2d26d6144e86" containerName="main" containerID="cri-o://3b63788c43df0038ebb1d18608140b9eb2e38f1d86b14ede8904951e779f821b" gracePeriod=45
Sep 11 12:39:54.231349 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.231322    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d" podUID="b502dcc5-5ffe-451f-9781-efcb69394137" containerName="main" containerID="cri-o://75e03e61aa9206fed9d049cf36cdef17ab23b58823978a54e4f6d8bc30183b55" gracePeriod=45
Sep 11 12:39:54.519425 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.519397    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:39:54.522442 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.522421    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:39:54.533839 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.533817    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:39:54.592911 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.592884    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb3646b9-acb4-43d6-8f30-768306fdb4b3-dshm\") pod \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\" (UID: \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\") "
Sep 11 12:39:54.593067 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.592922    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-tmp-dir\") pod \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\" (UID: \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\") "
Sep 11 12:39:54.593067 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.592951    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-tls-certs\") pod \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\" (UID: \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\") "
Sep 11 12:39:54.593067 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.592975    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb3646b9-acb4-43d6-8f30-768306fdb4b3-tls-certs\") pod \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\" (UID: \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\") "
Sep 11 12:39:54.593067 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.592998    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b502dcc5-5ffe-451f-9781-efcb69394137-home\") pod \"b502dcc5-5ffe-451f-9781-efcb69394137\" (UID: \"b502dcc5-5ffe-451f-9781-efcb69394137\") "
Sep 11 12:39:54.593067 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.593032    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dm5t\" (UniqueName: \"kubernetes.io/projected/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-kube-api-access-6dm5t\") pod \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\" (UID: \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\") "
Sep 11 12:39:54.593377 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.593226    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d5f9a4b-ce98-4594-ac66-2d26d6144e86" (UID: "0d5f9a4b-ce98-4594-ac66-2d26d6144e86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:39:54.593479 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.593455    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-home\") pod \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\" (UID: \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\") "
Sep 11 12:39:54.593479 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.593463    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b502dcc5-5ffe-451f-9781-efcb69394137-home" (OuterVolumeSpecName: "home") pod "b502dcc5-5ffe-451f-9781-efcb69394137" (UID: "b502dcc5-5ffe-451f-9781-efcb69394137"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:39:54.593604 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.593502    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b502dcc5-5ffe-451f-9781-efcb69394137-model-cache\") pod \"b502dcc5-5ffe-451f-9781-efcb69394137\" (UID: \"b502dcc5-5ffe-451f-9781-efcb69394137\") "
Sep 11 12:39:54.593604 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.593534    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-dshm\") pod \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\" (UID: \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\") "
Sep 11 12:39:54.593604 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.593563    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b502dcc5-5ffe-451f-9781-efcb69394137-dshm\") pod \"b502dcc5-5ffe-451f-9781-efcb69394137\" (UID: \"b502dcc5-5ffe-451f-9781-efcb69394137\") "
Sep 11 12:39:54.593604 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.593593    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-model-cache\") pod \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\" (UID: \"0d5f9a4b-ce98-4594-ac66-2d26d6144e86\") "
Sep 11 12:39:54.593818 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.593635    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b502dcc5-5ffe-451f-9781-efcb69394137-tls-certs\") pod \"b502dcc5-5ffe-451f-9781-efcb69394137\" (UID: \"b502dcc5-5ffe-451f-9781-efcb69394137\") "
Sep 11 12:39:54.593818 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.593688    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b502dcc5-5ffe-451f-9781-efcb69394137-tmp-dir\") pod \"b502dcc5-5ffe-451f-9781-efcb69394137\" (UID: \"b502dcc5-5ffe-451f-9781-efcb69394137\") "
Sep 11 12:39:54.593818 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.593692    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-home" (OuterVolumeSpecName: "home") pod "0d5f9a4b-ce98-4594-ac66-2d26d6144e86" (UID: "0d5f9a4b-ce98-4594-ac66-2d26d6144e86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:39:54.593818 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.593725    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb3646b9-acb4-43d6-8f30-768306fdb4b3-tmp-dir\") pod \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\" (UID: \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\") "
Sep 11 12:39:54.593818 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.593771    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gt68l\" (UniqueName: \"kubernetes.io/projected/b502dcc5-5ffe-451f-9781-efcb69394137-kube-api-access-gt68l\") pod \"b502dcc5-5ffe-451f-9781-efcb69394137\" (UID: \"b502dcc5-5ffe-451f-9781-efcb69394137\") "
Sep 11 12:39:54.593818 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.593795    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb3646b9-acb4-43d6-8f30-768306fdb4b3-model-cache\") pod \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\" (UID: \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\") "
Sep 11 12:39:54.594144 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.593839    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb3646b9-acb4-43d6-8f30-768306fdb4b3-home\") pod \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\" (UID: \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\") "
Sep 11 12:39:54.594144 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.593882    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d5f9a4b-ce98-4594-ac66-2d26d6144e86" (UID: "0d5f9a4b-ce98-4594-ac66-2d26d6144e86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:39:54.594144 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.593910    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7g7hv\" (UniqueName: \"kubernetes.io/projected/fb3646b9-acb4-43d6-8f30-768306fdb4b3-kube-api-access-7g7hv\") pod \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\" (UID: \"fb3646b9-acb4-43d6-8f30-768306fdb4b3\") "
Sep 11 12:39:54.594144 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.594074    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b502dcc5-5ffe-451f-9781-efcb69394137-model-cache" (OuterVolumeSpecName: "model-cache") pod "b502dcc5-5ffe-451f-9781-efcb69394137" (UID: "b502dcc5-5ffe-451f-9781-efcb69394137"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:39:54.594397 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.594326    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-tmp-dir\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:54.594397 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.594347    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b502dcc5-5ffe-451f-9781-efcb69394137-home\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:54.594397 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.594360    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-home\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:54.594397 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.594372    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b502dcc5-5ffe-451f-9781-efcb69394137-model-cache\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:54.594397 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.594385    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-model-cache\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:54.594665 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.594588    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb3646b9-acb4-43d6-8f30-768306fdb4b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fb3646b9-acb4-43d6-8f30-768306fdb4b3" (UID: "fb3646b9-acb4-43d6-8f30-768306fdb4b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:39:54.596181 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.596008    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b502dcc5-5ffe-451f-9781-efcb69394137-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b502dcc5-5ffe-451f-9781-efcb69394137" (UID: "b502dcc5-5ffe-451f-9781-efcb69394137"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:39:54.596602 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.596427    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb3646b9-acb4-43d6-8f30-768306fdb4b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "fb3646b9-acb4-43d6-8f30-768306fdb4b3" (UID: "fb3646b9-acb4-43d6-8f30-768306fdb4b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:39:54.596602 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.596484    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb3646b9-acb4-43d6-8f30-768306fdb4b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb3646b9-acb4-43d6-8f30-768306fdb4b3" (UID: "fb3646b9-acb4-43d6-8f30-768306fdb4b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:39:54.596763 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.596657    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-kube-api-access-6dm5t" (OuterVolumeSpecName: "kube-api-access-6dm5t") pod "0d5f9a4b-ce98-4594-ac66-2d26d6144e86" (UID: "0d5f9a4b-ce98-4594-ac66-2d26d6144e86"). InnerVolumeSpecName "kube-api-access-6dm5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:39:54.596938 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.596905    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb3646b9-acb4-43d6-8f30-768306fdb4b3-home" (OuterVolumeSpecName: "home") pod "fb3646b9-acb4-43d6-8f30-768306fdb4b3" (UID: "fb3646b9-acb4-43d6-8f30-768306fdb4b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:39:54.597072 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.597047    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb3646b9-acb4-43d6-8f30-768306fdb4b3-dshm" (OuterVolumeSpecName: "dshm") pod "fb3646b9-acb4-43d6-8f30-768306fdb4b3" (UID: "fb3646b9-acb4-43d6-8f30-768306fdb4b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:39:54.597151 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.597123    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d5f9a4b-ce98-4594-ac66-2d26d6144e86" (UID: "0d5f9a4b-ce98-4594-ac66-2d26d6144e86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:39:54.597347 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.597317    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b502dcc5-5ffe-451f-9781-efcb69394137-dshm" (OuterVolumeSpecName: "dshm") pod "b502dcc5-5ffe-451f-9781-efcb69394137" (UID: "b502dcc5-5ffe-451f-9781-efcb69394137"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:39:54.597517 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.597480    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-dshm" (OuterVolumeSpecName: "dshm") pod "0d5f9a4b-ce98-4594-ac66-2d26d6144e86" (UID: "0d5f9a4b-ce98-4594-ac66-2d26d6144e86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:39:54.597672 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.597648    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b502dcc5-5ffe-451f-9781-efcb69394137-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b502dcc5-5ffe-451f-9781-efcb69394137" (UID: "b502dcc5-5ffe-451f-9781-efcb69394137"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:39:54.598110 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.598090    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb3646b9-acb4-43d6-8f30-768306fdb4b3-kube-api-access-7g7hv" (OuterVolumeSpecName: "kube-api-access-7g7hv") pod "fb3646b9-acb4-43d6-8f30-768306fdb4b3" (UID: "fb3646b9-acb4-43d6-8f30-768306fdb4b3"). InnerVolumeSpecName "kube-api-access-7g7hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:39:54.598520 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.598501    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b502dcc5-5ffe-451f-9781-efcb69394137-kube-api-access-gt68l" (OuterVolumeSpecName: "kube-api-access-gt68l") pod "b502dcc5-5ffe-451f-9781-efcb69394137" (UID: "b502dcc5-5ffe-451f-9781-efcb69394137"). InnerVolumeSpecName "kube-api-access-gt68l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:39:54.695648 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.695610    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b502dcc5-5ffe-451f-9781-efcb69394137-tmp-dir\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:54.695648 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.695641    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb3646b9-acb4-43d6-8f30-768306fdb4b3-tmp-dir\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:54.695648 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.695650    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt68l\" (UniqueName: \"kubernetes.io/projected/b502dcc5-5ffe-451f-9781-efcb69394137-kube-api-access-gt68l\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:54.695878 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.695660    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb3646b9-acb4-43d6-8f30-768306fdb4b3-model-cache\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:54.695878 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.695669    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb3646b9-acb4-43d6-8f30-768306fdb4b3-home\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:54.695878 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.695679    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g7hv\" (UniqueName: \"kubernetes.io/projected/fb3646b9-acb4-43d6-8f30-768306fdb4b3-kube-api-access-7g7hv\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:54.695878 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.695687    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb3646b9-acb4-43d6-8f30-768306fdb4b3-dshm\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:54.695878 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.695695    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-tls-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:54.695878 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.695702    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb3646b9-acb4-43d6-8f30-768306fdb4b3-tls-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:54.695878 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.695710    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dm5t\" (UniqueName: \"kubernetes.io/projected/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-kube-api-access-6dm5t\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:54.695878 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.695717    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d5f9a4b-ce98-4594-ac66-2d26d6144e86-dshm\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:54.695878 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.695725    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b502dcc5-5ffe-451f-9781-efcb69394137-dshm\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:54.695878 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.695732    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b502dcc5-5ffe-451f-9781-efcb69394137-tls-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:39:54.932379 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:54.932300    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca" path="/var/lib/kubelet/pods/1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca/volumes"
Sep 11 12:39:55.035338 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.035305    2561 generic.go:356] "Generic (PLEG): container finished" podID="b502dcc5-5ffe-451f-9781-efcb69394137" containerID="75e03e61aa9206fed9d049cf36cdef17ab23b58823978a54e4f6d8bc30183b55" exitCode=0
Sep 11 12:39:55.035507 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.035345    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"
Sep 11 12:39:55.035507 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.035387    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d" event={"ID":"b502dcc5-5ffe-451f-9781-efcb69394137","Type":"ContainerDied","Data":"75e03e61aa9206fed9d049cf36cdef17ab23b58823978a54e4f6d8bc30183b55"}
Sep 11 12:39:55.035507 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.035430    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d" event={"ID":"b502dcc5-5ffe-451f-9781-efcb69394137","Type":"ContainerDied","Data":"7e27f9bd9464ddcf34d83a2b9a6472c88393043ce17402ba883f00562707f955"}
Sep 11 12:39:55.035507 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.035452    2561 scope.go:117] "RemoveContainer" containerID="75e03e61aa9206fed9d049cf36cdef17ab23b58823978a54e4f6d8bc30183b55"
Sep 11 12:39:55.036786 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.036763    2561 generic.go:356] "Generic (PLEG): container finished" podID="fb3646b9-acb4-43d6-8f30-768306fdb4b3" containerID="42983d8339d2c5a8b63244882b1cfaf484c6aec5eff91849a9714f19a38edd25" exitCode=0
Sep 11 12:39:55.036887 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.036808    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"
Sep 11 12:39:55.036887 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.036859    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58" event={"ID":"fb3646b9-acb4-43d6-8f30-768306fdb4b3","Type":"ContainerDied","Data":"42983d8339d2c5a8b63244882b1cfaf484c6aec5eff91849a9714f19a38edd25"}
Sep 11 12:39:55.036995 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.036888    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58" event={"ID":"fb3646b9-acb4-43d6-8f30-768306fdb4b3","Type":"ContainerDied","Data":"e00a223fcb70c91baa99172a23c12a661eacd568d4379d83a3ece41390e0b784"}
Sep 11 12:39:55.038670 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.038646    2561 generic.go:356] "Generic (PLEG): container finished" podID="0d5f9a4b-ce98-4594-ac66-2d26d6144e86" containerID="3b63788c43df0038ebb1d18608140b9eb2e38f1d86b14ede8904951e779f821b" exitCode=0
Sep 11 12:39:55.038768 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.038683    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"
Sep 11 12:39:55.038768 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.038696    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj" event={"ID":"0d5f9a4b-ce98-4594-ac66-2d26d6144e86","Type":"ContainerDied","Data":"3b63788c43df0038ebb1d18608140b9eb2e38f1d86b14ede8904951e779f821b"}
Sep 11 12:39:55.038768 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.038732    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj" event={"ID":"0d5f9a4b-ce98-4594-ac66-2d26d6144e86","Type":"ContainerDied","Data":"278d4dd9872f1c09b087052cff647f8b7dcb79d8f6eab489beaefd628ff8df67"}
Sep 11 12:39:55.045058 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.045043    2561 scope.go:117] "RemoveContainer" containerID="75e03e61aa9206fed9d049cf36cdef17ab23b58823978a54e4f6d8bc30183b55"
Sep 11 12:39:55.045348 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:39:55.045333    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75e03e61aa9206fed9d049cf36cdef17ab23b58823978a54e4f6d8bc30183b55\": container with ID starting with 75e03e61aa9206fed9d049cf36cdef17ab23b58823978a54e4f6d8bc30183b55 not found: ID does not exist" containerID="75e03e61aa9206fed9d049cf36cdef17ab23b58823978a54e4f6d8bc30183b55"
Sep 11 12:39:55.045401 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.045356    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75e03e61aa9206fed9d049cf36cdef17ab23b58823978a54e4f6d8bc30183b55"} err="failed to get container status \"75e03e61aa9206fed9d049cf36cdef17ab23b58823978a54e4f6d8bc30183b55\": rpc error: code = NotFound desc = could not find container \"75e03e61aa9206fed9d049cf36cdef17ab23b58823978a54e4f6d8bc30183b55\": container with ID starting with 75e03e61aa9206fed9d049cf36cdef17ab23b58823978a54e4f6d8bc30183b55 not found: ID does not exist"
Sep 11 12:39:55.045401 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.045372    2561 scope.go:117] "RemoveContainer" containerID="42983d8339d2c5a8b63244882b1cfaf484c6aec5eff91849a9714f19a38edd25"
Sep 11 12:39:55.055095 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.054070    2561 scope.go:117] "RemoveContainer" containerID="42983d8339d2c5a8b63244882b1cfaf484c6aec5eff91849a9714f19a38edd25"
Sep 11 12:39:55.055095 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:39:55.054867    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42983d8339d2c5a8b63244882b1cfaf484c6aec5eff91849a9714f19a38edd25\": container with ID starting with 42983d8339d2c5a8b63244882b1cfaf484c6aec5eff91849a9714f19a38edd25 not found: ID does not exist" containerID="42983d8339d2c5a8b63244882b1cfaf484c6aec5eff91849a9714f19a38edd25"
Sep 11 12:39:55.055095 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.054898    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42983d8339d2c5a8b63244882b1cfaf484c6aec5eff91849a9714f19a38edd25"} err="failed to get container status \"42983d8339d2c5a8b63244882b1cfaf484c6aec5eff91849a9714f19a38edd25\": rpc error: code = NotFound desc = could not find container \"42983d8339d2c5a8b63244882b1cfaf484c6aec5eff91849a9714f19a38edd25\": container with ID starting with 42983d8339d2c5a8b63244882b1cfaf484c6aec5eff91849a9714f19a38edd25 not found: ID does not exist"
Sep 11 12:39:55.055095 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.054922    2561 scope.go:117] "RemoveContainer" containerID="3b63788c43df0038ebb1d18608140b9eb2e38f1d86b14ede8904951e779f821b"
Sep 11 12:39:55.057440 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.057417    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"]
Sep 11 12:39:55.061147 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.061125    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/premium-w0-1a8ee3-kserve-77fbf9566c-b8dwj"]
Sep 11 12:39:55.066009 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.065993    2561 scope.go:117] "RemoveContainer" containerID="3b63788c43df0038ebb1d18608140b9eb2e38f1d86b14ede8904951e779f821b"
Sep 11 12:39:55.066285 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:39:55.066265    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b63788c43df0038ebb1d18608140b9eb2e38f1d86b14ede8904951e779f821b\": container with ID starting with 3b63788c43df0038ebb1d18608140b9eb2e38f1d86b14ede8904951e779f821b not found: ID does not exist" containerID="3b63788c43df0038ebb1d18608140b9eb2e38f1d86b14ede8904951e779f821b"
Sep 11 12:39:55.066380 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.066287    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b63788c43df0038ebb1d18608140b9eb2e38f1d86b14ede8904951e779f821b"} err="failed to get container status \"3b63788c43df0038ebb1d18608140b9eb2e38f1d86b14ede8904951e779f821b\": rpc error: code = NotFound desc = could not find container \"3b63788c43df0038ebb1d18608140b9eb2e38f1d86b14ede8904951e779f821b\": container with ID starting with 3b63788c43df0038ebb1d18608140b9eb2e38f1d86b14ede8904951e779f821b not found: ID does not exist"
Sep 11 12:39:55.071260 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.071237    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"]
Sep 11 12:39:55.076500 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.076480    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/simulator-w0-1a8ee3-kserve-558d6fb79d-69k5d"]
Sep 11 12:39:55.086593 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.086571    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"]
Sep 11 12:39:55.091127 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:55.091106    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/distinct-w0-1a8ee3-kserve-74995fc5d6-pzl58"]
Sep 11 12:39:56.932826 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:56.932793    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d5f9a4b-ce98-4594-ac66-2d26d6144e86" path="/var/lib/kubelet/pods/0d5f9a4b-ce98-4594-ac66-2d26d6144e86/volumes"
Sep 11 12:39:56.933225 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:56.933185    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b502dcc5-5ffe-451f-9781-efcb69394137" path="/var/lib/kubelet/pods/b502dcc5-5ffe-451f-9781-efcb69394137/volumes"
Sep 11 12:39:56.933607 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:56.933592    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb3646b9-acb4-43d6-8f30-768306fdb4b3" path="/var/lib/kubelet/pods/fb3646b9-acb4-43d6-8f30-768306fdb4b3/volumes"
Sep 11 12:39:59.580135 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:59.580095    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44f372-84bdj"]
Sep 11 12:39:59.580530 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:59.580478    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b502dcc5-5ffe-451f-9781-efcb69394137" containerName="main"
Sep 11 12:39:59.580530 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:59.580490    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b502dcc5-5ffe-451f-9781-efcb69394137" containerName="main"
Sep 11 12:39:59.580530 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:59.580498    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb3646b9-acb4-43d6-8f30-768306fdb4b3" containerName="main"
Sep 11 12:39:59.580530 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:59.580504    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb3646b9-acb4-43d6-8f30-768306fdb4b3" containerName="main"
Sep 11 12:39:59.580530 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:59.580514    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca" containerName="payload-pre-processing"
Sep 11 12:39:59.580530 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:59.580519    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca" containerName="payload-pre-processing"
Sep 11 12:39:59.580720 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:59.580536    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e102d7-70a9-46ac-af43-b3ef97ef6c74" containerName="revoke-keys"
Sep 11 12:39:59.580720 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:59.580541    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e102d7-70a9-46ac-af43-b3ef97ef6c74" containerName="revoke-keys"
Sep 11 12:39:59.580720 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:59.580555    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d5f9a4b-ce98-4594-ac66-2d26d6144e86" containerName="main"
Sep 11 12:39:59.580720 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:59.580560    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d5f9a4b-ce98-4594-ac66-2d26d6144e86" containerName="main"
Sep 11 12:39:59.580720 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:59.580613    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ffdd095-e1c7-4ec2-98ce-d0832c5ca0ca" containerName="payload-pre-processing"
Sep 11 12:39:59.580720 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:59.580623    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e102d7-70a9-46ac-af43-b3ef97ef6c74" containerName="revoke-keys"
Sep 11 12:39:59.580720 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:59.580629    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b502dcc5-5ffe-451f-9781-efcb69394137" containerName="main"
Sep 11 12:39:59.580720 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:59.580635    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d5f9a4b-ce98-4594-ac66-2d26d6144e86" containerName="main"
Sep 11 12:39:59.580720 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:59.580641    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb3646b9-acb4-43d6-8f30-768306fdb4b3" containerName="main"
Sep 11 12:39:59.583552 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:59.583536    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44f372-84bdj"
Sep 11 12:39:59.586004 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:59.585975    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 12:39:59.586132 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:59.586039    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\""
Sep 11 12:39:59.590871 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:59.590632    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44f372-84bdj"]
Sep 11 12:39:59.641104 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:59.641075    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0457b535-9e1d-4742-83fb-a8f75cf46a1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-44f372-84bdj\" (UID: \"0457b535-9e1d-4742-83fb-a8f75cf46a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44f372-84bdj"
Sep 11 12:39:59.742468 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:59.742431    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0457b535-9e1d-4742-83fb-a8f75cf46a1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-44f372-84bdj\" (UID: \"0457b535-9e1d-4742-83fb-a8f75cf46a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44f372-84bdj"
Sep 11 12:39:59.743053 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:59.743033    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0457b535-9e1d-4742-83fb-a8f75cf46a1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-44f372-84bdj\" (UID: \"0457b535-9e1d-4742-83fb-a8f75cf46a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44f372-84bdj"
Sep 11 12:39:59.894353 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:39:59.894269    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44f372-84bdj"
Sep 11 12:40:00.169601 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:00.169574    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44f372-84bdj"]
Sep 11 12:40:00.171502 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:40:00.171476    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0457b535_9e1d_4742_83fb_a8f75cf46a1e.slice/crio-ab8ff342a454b33838be087fed44a6821e6d3f6af66440f498460c14981269e7 WatchSource:0}: Error finding container ab8ff342a454b33838be087fed44a6821e6d3f6af66440f498460c14981269e7: Status 404 returned error can't find the container with id ab8ff342a454b33838be087fed44a6821e6d3f6af66440f498460c14981269e7
Sep 11 12:40:01.081066 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:01.081036    2561 generic.go:356] "Generic (PLEG): container finished" podID="0457b535-9e1d-4742-83fb-a8f75cf46a1e" containerID="63d7c3dfaac7ef7f0101500bf63fb663e48f2ca6eb96e46b9fd06fc83a170ef7" exitCode=0
Sep 11 12:40:01.081484 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:01.081129    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44f372-84bdj" event={"ID":"0457b535-9e1d-4742-83fb-a8f75cf46a1e","Type":"ContainerDied","Data":"63d7c3dfaac7ef7f0101500bf63fb663e48f2ca6eb96e46b9fd06fc83a170ef7"}
Sep 11 12:40:01.081484 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:01.081223    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44f372-84bdj" event={"ID":"0457b535-9e1d-4742-83fb-a8f75cf46a1e","Type":"ContainerStarted","Data":"ab8ff342a454b33838be087fed44a6821e6d3f6af66440f498460c14981269e7"}
Sep 11 12:40:02.228677 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:02.228654    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44f372-84bdj"
Sep 11 12:40:02.365306 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:02.365223    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0457b535-9e1d-4742-83fb-a8f75cf46a1e-maas-api-service-ca\") pod \"0457b535-9e1d-4742-83fb-a8f75cf46a1e\" (UID: \"0457b535-9e1d-4742-83fb-a8f75cf46a1e\") "
Sep 11 12:40:02.365574 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:02.365551    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0457b535-9e1d-4742-83fb-a8f75cf46a1e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0457b535-9e1d-4742-83fb-a8f75cf46a1e" (UID: "0457b535-9e1d-4742-83fb-a8f75cf46a1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:40:02.466593 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:02.466564    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0457b535-9e1d-4742-83fb-a8f75cf46a1e-maas-api-service-ca\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:40:03.092527 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:03.092498    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44f372-84bdj"
Sep 11 12:40:03.092527 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:03.092511    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44f372-84bdj" event={"ID":"0457b535-9e1d-4742-83fb-a8f75cf46a1e","Type":"ContainerDied","Data":"ab8ff342a454b33838be087fed44a6821e6d3f6af66440f498460c14981269e7"}
Sep 11 12:40:03.092731 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:03.092552    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab8ff342a454b33838be087fed44a6821e6d3f6af66440f498460c14981269e7"
Sep 11 12:40:09.227422 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:09.227392    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:40:10.173616 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:10.173584    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr"]
Sep 11 12:40:10.173895 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:10.173837    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr" podUID="8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778" containerName="payload-processing" containerID="cri-o://382ed57ad055da90383bb6e92da04891f64e7794699fd4a78c20d7146708bd3a" gracePeriod=30
Sep 11 12:40:10.336340 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:10.336305    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44f372-84bdj"]
Sep 11 12:40:10.338377 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:10.338351    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-44f372-84bdj"]
Sep 11 12:40:10.424958 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:10.424902    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr"
Sep 11 12:40:10.536450 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:10.536416    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdsj8\" (UniqueName: \"kubernetes.io/projected/8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778-kube-api-access-bdsj8\") pod \"8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778\" (UID: \"8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778\") "
Sep 11 12:40:10.536636 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:10.536479    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778-plugins-config-volume\") pod \"8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778\" (UID: \"8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778\") "
Sep 11 12:40:10.536857 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:10.536829    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778" (UID: "8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:40:10.538565 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:10.538543    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778-kube-api-access-bdsj8" (OuterVolumeSpecName: "kube-api-access-bdsj8") pod "8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778" (UID: "8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778"). InnerVolumeSpecName "kube-api-access-bdsj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:40:10.637544 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:10.637510    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdsj8\" (UniqueName: \"kubernetes.io/projected/8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778-kube-api-access-bdsj8\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:40:10.637544 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:10.637537    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:40:10.932550 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:10.932507    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0457b535-9e1d-4742-83fb-a8f75cf46a1e" path="/var/lib/kubelet/pods/0457b535-9e1d-4742-83fb-a8f75cf46a1e/volumes"
Sep 11 12:40:11.134536 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:11.134500    2561 generic.go:356] "Generic (PLEG): container finished" podID="8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778" containerID="382ed57ad055da90383bb6e92da04891f64e7794699fd4a78c20d7146708bd3a" exitCode=0
Sep 11 12:40:11.134709 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:11.134545    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr"
Sep 11 12:40:11.134709 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:11.134585    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr" event={"ID":"8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778","Type":"ContainerDied","Data":"382ed57ad055da90383bb6e92da04891f64e7794699fd4a78c20d7146708bd3a"}
Sep 11 12:40:11.134709 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:11.134645    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr" event={"ID":"8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778","Type":"ContainerDied","Data":"3ea7b12d674bb3adb30822c43015c7f6a1481be191f9414ba0d59152d58ccadd"}
Sep 11 12:40:11.134709 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:11.134667    2561 scope.go:117] "RemoveContainer" containerID="382ed57ad055da90383bb6e92da04891f64e7794699fd4a78c20d7146708bd3a"
Sep 11 12:40:11.143330 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:11.143310    2561 scope.go:117] "RemoveContainer" containerID="382ed57ad055da90383bb6e92da04891f64e7794699fd4a78c20d7146708bd3a"
Sep 11 12:40:11.143612 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:40:11.143595    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"382ed57ad055da90383bb6e92da04891f64e7794699fd4a78c20d7146708bd3a\": container with ID starting with 382ed57ad055da90383bb6e92da04891f64e7794699fd4a78c20d7146708bd3a not found: ID does not exist" containerID="382ed57ad055da90383bb6e92da04891f64e7794699fd4a78c20d7146708bd3a"
Sep 11 12:40:11.143657 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:11.143619    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"382ed57ad055da90383bb6e92da04891f64e7794699fd4a78c20d7146708bd3a"} err="failed to get container status \"382ed57ad055da90383bb6e92da04891f64e7794699fd4a78c20d7146708bd3a\": rpc error: code = NotFound desc = could not find container \"382ed57ad055da90383bb6e92da04891f64e7794699fd4a78c20d7146708bd3a\": container with ID starting with 382ed57ad055da90383bb6e92da04891f64e7794699fd4a78c20d7146708bd3a not found: ID does not exist"
Sep 11 12:40:11.150962 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:11.150940    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr"]
Sep 11 12:40:11.156433 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:11.156414    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-44f372-548b77564f-mrqjr"]
Sep 11 12:40:12.933076 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:12.933044    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778" path="/var/lib/kubelet/pods/8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778/volumes"
Sep 11 12:40:17.894715 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:17.894678    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a50cf9-kf28h"]
Sep 11 12:40:17.895089 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:17.895075    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0457b535-9e1d-4742-83fb-a8f75cf46a1e" containerName="revoke-keys"
Sep 11 12:40:17.895089 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:17.895090    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0457b535-9e1d-4742-83fb-a8f75cf46a1e" containerName="revoke-keys"
Sep 11 12:40:17.895179 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:17.895121    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778" containerName="payload-processing"
Sep 11 12:40:17.895179 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:17.895130    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778" containerName="payload-processing"
Sep 11 12:40:17.895284 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:17.895206    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0457b535-9e1d-4742-83fb-a8f75cf46a1e" containerName="revoke-keys"
Sep 11 12:40:17.895284 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:17.895220    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aad8178-8fe7-4b6d-9cb4-5ec1fc3ac778" containerName="payload-processing"
Sep 11 12:40:17.899551 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:17.899534    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a50cf9-kf28h"
Sep 11 12:40:17.902241 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:17.902222    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 12:40:17.902357 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:17.902227    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\""
Sep 11 12:40:17.905956 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:17.905935    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a50cf9-kf28h"]
Sep 11 12:40:18.005016 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:18.004983    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/663a98dc-9b2b-43f5-9134-faa0e4264574-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a50cf9-kf28h\" (UID: \"663a98dc-9b2b-43f5-9134-faa0e4264574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a50cf9-kf28h"
Sep 11 12:40:18.105829 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:18.105795    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/663a98dc-9b2b-43f5-9134-faa0e4264574-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a50cf9-kf28h\" (UID: \"663a98dc-9b2b-43f5-9134-faa0e4264574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a50cf9-kf28h"
Sep 11 12:40:18.106481 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:18.106462    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/663a98dc-9b2b-43f5-9134-faa0e4264574-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a50cf9-kf28h\" (UID: \"663a98dc-9b2b-43f5-9134-faa0e4264574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a50cf9-kf28h"
Sep 11 12:40:18.210327 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:18.210253    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a50cf9-kf28h"
Sep 11 12:40:18.480415 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:18.480389    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a50cf9-kf28h"]
Sep 11 12:40:18.482123 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:40:18.482096    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663a98dc_9b2b_43f5_9134_faa0e4264574.slice/crio-94e347dd415745337fb5bc4a7269bf249bbc122d7882b0277316116c5b0bd559 WatchSource:0}: Error finding container 94e347dd415745337fb5bc4a7269bf249bbc122d7882b0277316116c5b0bd559: Status 404 returned error can't find the container with id 94e347dd415745337fb5bc4a7269bf249bbc122d7882b0277316116c5b0bd559
Sep 11 12:40:19.179416 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:19.179389    2561 generic.go:356] "Generic (PLEG): container finished" podID="663a98dc-9b2b-43f5-9134-faa0e4264574" containerID="c5b6b56b342cdc331a90f26577aff9098bb948ef72cf3ed19c2a743947fd5d69" exitCode=0
Sep 11 12:40:19.179754 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:19.179453    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a50cf9-kf28h" event={"ID":"663a98dc-9b2b-43f5-9134-faa0e4264574","Type":"ContainerDied","Data":"c5b6b56b342cdc331a90f26577aff9098bb948ef72cf3ed19c2a743947fd5d69"}
Sep 11 12:40:19.179754 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:19.179513    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a50cf9-kf28h" event={"ID":"663a98dc-9b2b-43f5-9134-faa0e4264574","Type":"ContainerStarted","Data":"94e347dd415745337fb5bc4a7269bf249bbc122d7882b0277316116c5b0bd559"}
Sep 11 12:40:20.318021 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:20.317996    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a50cf9-kf28h"
Sep 11 12:40:20.426704 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:20.426672    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/663a98dc-9b2b-43f5-9134-faa0e4264574-maas-api-service-ca\") pod \"663a98dc-9b2b-43f5-9134-faa0e4264574\" (UID: \"663a98dc-9b2b-43f5-9134-faa0e4264574\") "
Sep 11 12:40:20.427003 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:20.426981    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663a98dc-9b2b-43f5-9134-faa0e4264574-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "663a98dc-9b2b-43f5-9134-faa0e4264574" (UID: "663a98dc-9b2b-43f5-9134-faa0e4264574"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:40:20.528015 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:20.527991    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/663a98dc-9b2b-43f5-9134-faa0e4264574-maas-api-service-ca\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:40:21.191603 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:21.191518    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a50cf9-kf28h"
Sep 11 12:40:21.191603 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:21.191560    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a50cf9-kf28h" event={"ID":"663a98dc-9b2b-43f5-9134-faa0e4264574","Type":"ContainerDied","Data":"94e347dd415745337fb5bc4a7269bf249bbc122d7882b0277316116c5b0bd559"}
Sep 11 12:40:21.191782 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:21.191612    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94e347dd415745337fb5bc4a7269bf249bbc122d7882b0277316116c5b0bd559"
Sep 11 12:40:28.753164 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:28.753132    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq"]
Sep 11 12:40:28.753667 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:28.753375    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq" podUID="630ceb15-2cf6-4268-9e8e-27d6c2530fc7" containerName="payload-processing" containerID="cri-o://3bf3d439e0a8420626068caf3e57acee8fea2cc1e408bb8d4eee8e21408799fd" gracePeriod=30
Sep 11 12:40:28.762714 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:28.762673    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt"]
Sep 11 12:40:28.763205 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:28.763137    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt" podUID="6d179607-e322-4187-86d5-e3af6a0b1b60" containerName="payload-pre-processing" containerID="cri-o://0d9b0ac847f5f826faedeaf2a3d07855ccef08ac7d8bf5b82f43340e188ca027" gracePeriod=30
Sep 11 12:40:28.912772 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:28.912739    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a50cf9-kf28h"]
Sep 11 12:40:28.915489 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:28.915463    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a50cf9-kf28h"]
Sep 11 12:40:28.935656 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:28.935623    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="663a98dc-9b2b-43f5-9134-faa0e4264574" path="/var/lib/kubelet/pods/663a98dc-9b2b-43f5-9134-faa0e4264574/volumes"
Sep 11 12:40:29.042862 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.042786    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq"
Sep 11 12:40:29.062324 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.044823    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt"
Sep 11 12:40:29.083942 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.083919    2561 scope.go:117] "RemoveContainer" containerID="3bf3d439e0a8420626068caf3e57acee8fea2cc1e408bb8d4eee8e21408799fd"
Sep 11 12:40:29.094172 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.094152    2561 scope.go:117] "RemoveContainer" containerID="0d9b0ac847f5f826faedeaf2a3d07855ccef08ac7d8bf5b82f43340e188ca027"
Sep 11 12:40:29.113302 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.113265    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwzqr\" (UniqueName: \"kubernetes.io/projected/6d179607-e322-4187-86d5-e3af6a0b1b60-kube-api-access-mwzqr\") pod \"6d179607-e322-4187-86d5-e3af6a0b1b60\" (UID: \"6d179607-e322-4187-86d5-e3af6a0b1b60\") "
Sep 11 12:40:29.113413 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.113313    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d179607-e322-4187-86d5-e3af6a0b1b60-plugins-config-volume\") pod \"6d179607-e322-4187-86d5-e3af6a0b1b60\" (UID: \"6d179607-e322-4187-86d5-e3af6a0b1b60\") "
Sep 11 12:40:29.113413 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.113391    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/630ceb15-2cf6-4268-9e8e-27d6c2530fc7-plugins-config-volume\") pod \"630ceb15-2cf6-4268-9e8e-27d6c2530fc7\" (UID: \"630ceb15-2cf6-4268-9e8e-27d6c2530fc7\") "
Sep 11 12:40:29.113542 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.113449    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4dcq\" (UniqueName: \"kubernetes.io/projected/630ceb15-2cf6-4268-9e8e-27d6c2530fc7-kube-api-access-r4dcq\") pod \"630ceb15-2cf6-4268-9e8e-27d6c2530fc7\" (UID: \"630ceb15-2cf6-4268-9e8e-27d6c2530fc7\") "
Sep 11 12:40:29.113924 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.113868    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/630ceb15-2cf6-4268-9e8e-27d6c2530fc7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "630ceb15-2cf6-4268-9e8e-27d6c2530fc7" (UID: "630ceb15-2cf6-4268-9e8e-27d6c2530fc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:40:29.114045 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.114019    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d179607-e322-4187-86d5-e3af6a0b1b60-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6d179607-e322-4187-86d5-e3af6a0b1b60" (UID: "6d179607-e322-4187-86d5-e3af6a0b1b60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:40:29.115590 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.115563    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d179607-e322-4187-86d5-e3af6a0b1b60-kube-api-access-mwzqr" (OuterVolumeSpecName: "kube-api-access-mwzqr") pod "6d179607-e322-4187-86d5-e3af6a0b1b60" (UID: "6d179607-e322-4187-86d5-e3af6a0b1b60"). InnerVolumeSpecName "kube-api-access-mwzqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:40:29.115690 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.115670    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630ceb15-2cf6-4268-9e8e-27d6c2530fc7-kube-api-access-r4dcq" (OuterVolumeSpecName: "kube-api-access-r4dcq") pod "630ceb15-2cf6-4268-9e8e-27d6c2530fc7" (UID: "630ceb15-2cf6-4268-9e8e-27d6c2530fc7"). InnerVolumeSpecName "kube-api-access-r4dcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:40:29.214530 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.214502    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/630ceb15-2cf6-4268-9e8e-27d6c2530fc7-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:40:29.214530 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.214525    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4dcq\" (UniqueName: \"kubernetes.io/projected/630ceb15-2cf6-4268-9e8e-27d6c2530fc7-kube-api-access-r4dcq\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:40:29.214667 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.214539    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwzqr\" (UniqueName: \"kubernetes.io/projected/6d179607-e322-4187-86d5-e3af6a0b1b60-kube-api-access-mwzqr\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:40:29.214667 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.214552    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d179607-e322-4187-86d5-e3af6a0b1b60-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:40:29.232508 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.232474    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt" event={"ID":"6d179607-e322-4187-86d5-e3af6a0b1b60","Type":"ContainerDied","Data":"0d9b0ac847f5f826faedeaf2a3d07855ccef08ac7d8bf5b82f43340e188ca027"}
Sep 11 12:40:29.232623 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.232534    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt"
Sep 11 12:40:29.232623 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.232540    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt" event={"ID":"6d179607-e322-4187-86d5-e3af6a0b1b60","Type":"ContainerDied","Data":"2cbca824da889a864ab25d3d15a4079305fad2c6ec6aeeda1cb57f954c798af3"}
Sep 11 12:40:29.232623 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.232545    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq"
Sep 11 12:40:29.232623 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.232564    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq" event={"ID":"630ceb15-2cf6-4268-9e8e-27d6c2530fc7","Type":"ContainerDied","Data":"3bf3d439e0a8420626068caf3e57acee8fea2cc1e408bb8d4eee8e21408799fd"}
Sep 11 12:40:29.232623 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.232590    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq" event={"ID":"630ceb15-2cf6-4268-9e8e-27d6c2530fc7","Type":"ContainerDied","Data":"d97f1f233e1738278dc1c9119a89615ae7a4e39887b3c1dc124c6eed3b36add6"}
Sep 11 12:40:29.262058 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.262007    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt"]
Sep 11 12:40:29.267717 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.267690    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a50cf9-6f8bffb8fd-cblgt"]
Sep 11 12:40:29.295032 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.295003    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq"]
Sep 11 12:40:29.298942 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:29.298918    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a50cf9-76ccc74df9-vlpdq"]
Sep 11 12:40:30.932688 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:30.932651    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="630ceb15-2cf6-4268-9e8e-27d6c2530fc7" path="/var/lib/kubelet/pods/630ceb15-2cf6-4268-9e8e-27d6c2530fc7/volumes"
Sep 11 12:40:30.933040 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:30.933006    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d179607-e322-4187-86d5-e3af6a0b1b60" path="/var/lib/kubelet/pods/6d179607-e322-4187-86d5-e3af6a0b1b60/volumes"
Sep 11 12:40:35.026581 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:35.026539    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:40:42.257026 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:42.256993    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cb20c8-2gs55"]
Sep 11 12:40:42.257408 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:42.257365    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="663a98dc-9b2b-43f5-9134-faa0e4264574" containerName="revoke-keys"
Sep 11 12:40:42.257408 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:42.257379    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="663a98dc-9b2b-43f5-9134-faa0e4264574" containerName="revoke-keys"
Sep 11 12:40:42.257408 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:42.257385    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d179607-e322-4187-86d5-e3af6a0b1b60" containerName="payload-pre-processing"
Sep 11 12:40:42.257408 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:42.257390    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d179607-e322-4187-86d5-e3af6a0b1b60" containerName="payload-pre-processing"
Sep 11 12:40:42.257408 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:42.257403    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="630ceb15-2cf6-4268-9e8e-27d6c2530fc7" containerName="payload-processing"
Sep 11 12:40:42.257408 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:42.257408    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="630ceb15-2cf6-4268-9e8e-27d6c2530fc7" containerName="payload-processing"
Sep 11 12:40:42.257604 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:42.257460    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="663a98dc-9b2b-43f5-9134-faa0e4264574" containerName="revoke-keys"
Sep 11 12:40:42.257604 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:42.257468    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d179607-e322-4187-86d5-e3af6a0b1b60" containerName="payload-pre-processing"
Sep 11 12:40:42.257604 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:42.257477    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="630ceb15-2cf6-4268-9e8e-27d6c2530fc7" containerName="payload-processing"
Sep 11 12:40:42.261756 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:42.261739    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cb20c8-2gs55"
Sep 11 12:40:42.264276 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:42.264253    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 12:40:42.264401 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:42.264257    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\""
Sep 11 12:40:42.267954 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:42.267930    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cb20c8-2gs55"]
Sep 11 12:40:42.343469 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:42.343439    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07822040-fed3-4d3e-a34e-e43a30bbf94b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-cb20c8-2gs55\" (UID: \"07822040-fed3-4d3e-a34e-e43a30bbf94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cb20c8-2gs55"
Sep 11 12:40:42.444256 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:42.444229    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07822040-fed3-4d3e-a34e-e43a30bbf94b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-cb20c8-2gs55\" (UID: \"07822040-fed3-4d3e-a34e-e43a30bbf94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cb20c8-2gs55"
Sep 11 12:40:42.444789 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:42.444771    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07822040-fed3-4d3e-a34e-e43a30bbf94b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-cb20c8-2gs55\" (UID: \"07822040-fed3-4d3e-a34e-e43a30bbf94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cb20c8-2gs55"
Sep 11 12:40:42.574160 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:42.574089    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cb20c8-2gs55"
Sep 11 12:40:42.844392 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:42.844311    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cb20c8-2gs55"]
Sep 11 12:40:42.847983 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:40:42.847953    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07822040_fed3_4d3e_a34e_e43a30bbf94b.slice/crio-5e3cd8af666163d2ee6a3166a61366169ab4905f7a4d0cb644299de1a9301487 WatchSource:0}: Error finding container 5e3cd8af666163d2ee6a3166a61366169ab4905f7a4d0cb644299de1a9301487: Status 404 returned error can't find the container with id 5e3cd8af666163d2ee6a3166a61366169ab4905f7a4d0cb644299de1a9301487
Sep 11 12:40:43.299494 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:43.299461    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cb20c8-2gs55" event={"ID":"07822040-fed3-4d3e-a34e-e43a30bbf94b","Type":"ContainerStarted","Data":"7dc1b2b76b4ada6232be46e47d2088ce914c5c5d4c7aa700c2bd98b46a192612"}
Sep 11 12:40:43.299819 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:43.299539    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cb20c8-2gs55" event={"ID":"07822040-fed3-4d3e-a34e-e43a30bbf94b","Type":"ContainerStarted","Data":"5e3cd8af666163d2ee6a3166a61366169ab4905f7a4d0cb644299de1a9301487"}
Sep 11 12:40:43.315127 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:43.315079    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cb20c8-2gs55" podStartSLOduration=1.315065345 podStartE2EDuration="1.315065345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:40:43.314104649 +0000 UTC m=+1574.876884412" watchObservedRunningTime="2026-09-11 12:40:43.315065345 +0000 UTC m=+1574.877845108"
Sep 11 12:40:44.306416 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:44.306380    2561 generic.go:356] "Generic (PLEG): container finished" podID="07822040-fed3-4d3e-a34e-e43a30bbf94b" containerID="7dc1b2b76b4ada6232be46e47d2088ce914c5c5d4c7aa700c2bd98b46a192612" exitCode=0
Sep 11 12:40:44.306784 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:44.306423    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cb20c8-2gs55" event={"ID":"07822040-fed3-4d3e-a34e-e43a30bbf94b","Type":"ContainerDied","Data":"7dc1b2b76b4ada6232be46e47d2088ce914c5c5d4c7aa700c2bd98b46a192612"}
Sep 11 12:40:45.454555 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:45.454533    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cb20c8-2gs55"
Sep 11 12:40:45.571803 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:45.571713    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07822040-fed3-4d3e-a34e-e43a30bbf94b-maas-api-service-ca\") pod \"07822040-fed3-4d3e-a34e-e43a30bbf94b\" (UID: \"07822040-fed3-4d3e-a34e-e43a30bbf94b\") "
Sep 11 12:40:45.572051 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:45.572028    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07822040-fed3-4d3e-a34e-e43a30bbf94b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "07822040-fed3-4d3e-a34e-e43a30bbf94b" (UID: "07822040-fed3-4d3e-a34e-e43a30bbf94b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:40:45.673061 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:45.673036    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07822040-fed3-4d3e-a34e-e43a30bbf94b-maas-api-service-ca\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:40:46.317829 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:46.317803    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cb20c8-2gs55"
Sep 11 12:40:46.317998 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:46.317808    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cb20c8-2gs55" event={"ID":"07822040-fed3-4d3e-a34e-e43a30bbf94b","Type":"ContainerDied","Data":"5e3cd8af666163d2ee6a3166a61366169ab4905f7a4d0cb644299de1a9301487"}
Sep 11 12:40:46.317998 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:46.317915    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e3cd8af666163d2ee6a3166a61366169ab4905f7a4d0cb644299de1a9301487"
Sep 11 12:40:47.118893 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:47.118863    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"]
Sep 11 12:40:47.148383 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:47.148354    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"]
Sep 11 12:40:56.030094 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:56.030052    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:40:58.035313 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.035277    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5"]
Sep 11 12:40:58.039029 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.039000    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk"]
Sep 11 12:40:58.039240 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.039213    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk" podUID="6d2a4d3c-95d8-4dec-b576-9ef468ffcf42" containerName="payload-pre-processing" containerID="cri-o://b0b90dbca3f9fb27a6b87a5d3655df9d85a2e8541fa74e0f2933289f46eeacf2" gracePeriod=30
Sep 11 12:40:58.264022 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.263990    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cb20c8-2gs55"]
Sep 11 12:40:58.272274 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.272246    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cb20c8-2gs55"]
Sep 11 12:40:58.287821 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.287765    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk"
Sep 11 12:40:58.374081 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.374051    2561 generic.go:356] "Generic (PLEG): container finished" podID="6d2a4d3c-95d8-4dec-b576-9ef468ffcf42" containerID="b0b90dbca3f9fb27a6b87a5d3655df9d85a2e8541fa74e0f2933289f46eeacf2" exitCode=0
Sep 11 12:40:58.374249 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.374094    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk"
Sep 11 12:40:58.374249 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.374138    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk" event={"ID":"6d2a4d3c-95d8-4dec-b576-9ef468ffcf42","Type":"ContainerDied","Data":"b0b90dbca3f9fb27a6b87a5d3655df9d85a2e8541fa74e0f2933289f46eeacf2"}
Sep 11 12:40:58.374249 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.374199    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk" event={"ID":"6d2a4d3c-95d8-4dec-b576-9ef468ffcf42","Type":"ContainerDied","Data":"24974b293c3939fa40fd94f0be34396dee5b337c25470035427edfeaa69835e0"}
Sep 11 12:40:58.374249 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.374223    2561 scope.go:117] "RemoveContainer" containerID="b0b90dbca3f9fb27a6b87a5d3655df9d85a2e8541fa74e0f2933289f46eeacf2"
Sep 11 12:40:58.374496 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.374385    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5" podUID="3f49fb2d-fa85-4cba-a555-6e455299cb06" containerName="payload-processing" containerID="cri-o://4cefa318e39077adf21a703fe5efb4e5cfc3f51aa0e91059ba98e2c8427afee2" gracePeriod=30
Sep 11 12:40:58.383145 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.383125    2561 scope.go:117] "RemoveContainer" containerID="b0b90dbca3f9fb27a6b87a5d3655df9d85a2e8541fa74e0f2933289f46eeacf2"
Sep 11 12:40:58.383473 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:40:58.383454    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0b90dbca3f9fb27a6b87a5d3655df9d85a2e8541fa74e0f2933289f46eeacf2\": container with ID starting with b0b90dbca3f9fb27a6b87a5d3655df9d85a2e8541fa74e0f2933289f46eeacf2 not found: ID does not exist" containerID="b0b90dbca3f9fb27a6b87a5d3655df9d85a2e8541fa74e0f2933289f46eeacf2"
Sep 11 12:40:58.383536 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.383481    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0b90dbca3f9fb27a6b87a5d3655df9d85a2e8541fa74e0f2933289f46eeacf2"} err="failed to get container status \"b0b90dbca3f9fb27a6b87a5d3655df9d85a2e8541fa74e0f2933289f46eeacf2\": rpc error: code = NotFound desc = could not find container \"b0b90dbca3f9fb27a6b87a5d3655df9d85a2e8541fa74e0f2933289f46eeacf2\": container with ID starting with b0b90dbca3f9fb27a6b87a5d3655df9d85a2e8541fa74e0f2933289f46eeacf2 not found: ID does not exist"
Sep 11 12:40:58.388866 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.388845    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xc6bq\" (UniqueName: \"kubernetes.io/projected/6d2a4d3c-95d8-4dec-b576-9ef468ffcf42-kube-api-access-xc6bq\") pod \"6d2a4d3c-95d8-4dec-b576-9ef468ffcf42\" (UID: \"6d2a4d3c-95d8-4dec-b576-9ef468ffcf42\") "
Sep 11 12:40:58.388939 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.388906    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2a4d3c-95d8-4dec-b576-9ef468ffcf42-plugins-config-volume\") pod \"6d2a4d3c-95d8-4dec-b576-9ef468ffcf42\" (UID: \"6d2a4d3c-95d8-4dec-b576-9ef468ffcf42\") "
Sep 11 12:40:58.389340 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.389287    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d2a4d3c-95d8-4dec-b576-9ef468ffcf42-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6d2a4d3c-95d8-4dec-b576-9ef468ffcf42" (UID: "6d2a4d3c-95d8-4dec-b576-9ef468ffcf42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:40:58.390849 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.390826    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d2a4d3c-95d8-4dec-b576-9ef468ffcf42-kube-api-access-xc6bq" (OuterVolumeSpecName: "kube-api-access-xc6bq") pod "6d2a4d3c-95d8-4dec-b576-9ef468ffcf42" (UID: "6d2a4d3c-95d8-4dec-b576-9ef468ffcf42"). InnerVolumeSpecName "kube-api-access-xc6bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:40:58.490118 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.490087    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc6bq\" (UniqueName: \"kubernetes.io/projected/6d2a4d3c-95d8-4dec-b576-9ef468ffcf42-kube-api-access-xc6bq\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:40:58.490118 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.490114    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2a4d3c-95d8-4dec-b576-9ef468ffcf42-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:40:58.611828 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.611802    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5"
Sep 11 12:40:58.695494 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.695468    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk"]
Sep 11 12:40:58.699411 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.699389    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-cb20c8-5d9b544d94-j6znk"]
Sep 11 12:40:58.793496 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.793418    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79hkj\" (UniqueName: \"kubernetes.io/projected/3f49fb2d-fa85-4cba-a555-6e455299cb06-kube-api-access-79hkj\") pod \"3f49fb2d-fa85-4cba-a555-6e455299cb06\" (UID: \"3f49fb2d-fa85-4cba-a555-6e455299cb06\") "
Sep 11 12:40:58.793496 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.793471    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f49fb2d-fa85-4cba-a555-6e455299cb06-plugins-config-volume\") pod \"3f49fb2d-fa85-4cba-a555-6e455299cb06\" (UID: \"3f49fb2d-fa85-4cba-a555-6e455299cb06\") "
Sep 11 12:40:58.793857 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.793835    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f49fb2d-fa85-4cba-a555-6e455299cb06-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3f49fb2d-fa85-4cba-a555-6e455299cb06" (UID: "3f49fb2d-fa85-4cba-a555-6e455299cb06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:40:58.795552 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.795532    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f49fb2d-fa85-4cba-a555-6e455299cb06-kube-api-access-79hkj" (OuterVolumeSpecName: "kube-api-access-79hkj") pod "3f49fb2d-fa85-4cba-a555-6e455299cb06" (UID: "3f49fb2d-fa85-4cba-a555-6e455299cb06"). InnerVolumeSpecName "kube-api-access-79hkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:40:58.894275 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.894239    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79hkj\" (UniqueName: \"kubernetes.io/projected/3f49fb2d-fa85-4cba-a555-6e455299cb06-kube-api-access-79hkj\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:40:58.894275 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.894272    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f49fb2d-fa85-4cba-a555-6e455299cb06-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:40:58.932028 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.932000    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07822040-fed3-4d3e-a34e-e43a30bbf94b" path="/var/lib/kubelet/pods/07822040-fed3-4d3e-a34e-e43a30bbf94b/volumes"
Sep 11 12:40:58.932402 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:58.932389    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d2a4d3c-95d8-4dec-b576-9ef468ffcf42" path="/var/lib/kubelet/pods/6d2a4d3c-95d8-4dec-b576-9ef468ffcf42/volumes"
Sep 11 12:40:59.379669 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:59.379573    2561 generic.go:356] "Generic (PLEG): container finished" podID="3f49fb2d-fa85-4cba-a555-6e455299cb06" containerID="4cefa318e39077adf21a703fe5efb4e5cfc3f51aa0e91059ba98e2c8427afee2" exitCode=0
Sep 11 12:40:59.379669 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:59.379617    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5"
Sep 11 12:40:59.379669 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:59.379655    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5" event={"ID":"3f49fb2d-fa85-4cba-a555-6e455299cb06","Type":"ContainerDied","Data":"4cefa318e39077adf21a703fe5efb4e5cfc3f51aa0e91059ba98e2c8427afee2"}
Sep 11 12:40:59.380255 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:59.379719    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5" event={"ID":"3f49fb2d-fa85-4cba-a555-6e455299cb06","Type":"ContainerDied","Data":"0c6b4d1f66795eaad9fae0d9f7db044263ebe164e44f9fd64cbd5fb2ca2c63e1"}
Sep 11 12:40:59.380255 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:59.379747    2561 scope.go:117] "RemoveContainer" containerID="4cefa318e39077adf21a703fe5efb4e5cfc3f51aa0e91059ba98e2c8427afee2"
Sep 11 12:40:59.388478 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:59.388460    2561 scope.go:117] "RemoveContainer" containerID="4cefa318e39077adf21a703fe5efb4e5cfc3f51aa0e91059ba98e2c8427afee2"
Sep 11 12:40:59.388724 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:40:59.388708    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cefa318e39077adf21a703fe5efb4e5cfc3f51aa0e91059ba98e2c8427afee2\": container with ID starting with 4cefa318e39077adf21a703fe5efb4e5cfc3f51aa0e91059ba98e2c8427afee2 not found: ID does not exist" containerID="4cefa318e39077adf21a703fe5efb4e5cfc3f51aa0e91059ba98e2c8427afee2"
Sep 11 12:40:59.388771 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:59.388733    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cefa318e39077adf21a703fe5efb4e5cfc3f51aa0e91059ba98e2c8427afee2"} err="failed to get container status \"4cefa318e39077adf21a703fe5efb4e5cfc3f51aa0e91059ba98e2c8427afee2\": rpc error: code = NotFound desc = could not find container \"4cefa318e39077adf21a703fe5efb4e5cfc3f51aa0e91059ba98e2c8427afee2\": container with ID starting with 4cefa318e39077adf21a703fe5efb4e5cfc3f51aa0e91059ba98e2c8427afee2 not found: ID does not exist"
Sep 11 12:40:59.396911 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:59.396881    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5"]
Sep 11 12:40:59.402665 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:40:59.402645    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cb20c8-7866456f4c-lqvm5"]
Sep 11 12:41:00.932481 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:00.932452    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f49fb2d-fa85-4cba-a555-6e455299cb06" path="/var/lib/kubelet/pods/3f49fb2d-fa85-4cba-a555-6e455299cb06/volumes"
Sep 11 12:41:02.138000 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.137955    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm" podUID="92ad8452-ea85-436c-824c-e270e48a9c39" containerName="main" containerID="cri-o://772daab48494e55059cfc087aa2b56e480442f9c762313c9b95d10d8e87257ce" gracePeriod=45
Sep 11 12:41:02.166622 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.166589    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n" podUID="5f6731d4-62c8-4e59-a6be-acc6a9ae6082" containerName="main" containerID="cri-o://efbcc68072ab74d2e00b15a8e42ccf0bc013341d468966807100f1b2a4e0cb1c" gracePeriod=45
Sep 11 12:41:02.404264 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.404237    2561 generic.go:356] "Generic (PLEG): container finished" podID="5f6731d4-62c8-4e59-a6be-acc6a9ae6082" containerID="efbcc68072ab74d2e00b15a8e42ccf0bc013341d468966807100f1b2a4e0cb1c" exitCode=0
Sep 11 12:41:02.404405 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.404314    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n" event={"ID":"5f6731d4-62c8-4e59-a6be-acc6a9ae6082","Type":"ContainerDied","Data":"efbcc68072ab74d2e00b15a8e42ccf0bc013341d468966807100f1b2a4e0cb1c"}
Sep 11 12:41:02.405874 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.405847    2561 generic.go:356] "Generic (PLEG): container finished" podID="92ad8452-ea85-436c-824c-e270e48a9c39" containerID="772daab48494e55059cfc087aa2b56e480442f9c762313c9b95d10d8e87257ce" exitCode=0
Sep 11 12:41:02.405971 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.405912    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm" event={"ID":"92ad8452-ea85-436c-824c-e270e48a9c39","Type":"ContainerDied","Data":"772daab48494e55059cfc087aa2b56e480442f9c762313c9b95d10d8e87257ce"}
Sep 11 12:41:02.428358 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.428332    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:41:02.432148 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.432126    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:41:02.527610 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.527579    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92ad8452-ea85-436c-824c-e270e48a9c39-home\") pod \"92ad8452-ea85-436c-824c-e270e48a9c39\" (UID: \"92ad8452-ea85-436c-824c-e270e48a9c39\") "
Sep 11 12:41:02.527788 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.527620    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-home\") pod \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\" (UID: \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\") "
Sep 11 12:41:02.527788 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.527648    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92ad8452-ea85-436c-824c-e270e48a9c39-tmp-dir\") pod \"92ad8452-ea85-436c-824c-e270e48a9c39\" (UID: \"92ad8452-ea85-436c-824c-e270e48a9c39\") "
Sep 11 12:41:02.527788 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.527668    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv9mk\" (UniqueName: \"kubernetes.io/projected/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-kube-api-access-mv9mk\") pod \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\" (UID: \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\") "
Sep 11 12:41:02.527788 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.527693    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92ad8452-ea85-436c-824c-e270e48a9c39-tls-certs\") pod \"92ad8452-ea85-436c-824c-e270e48a9c39\" (UID: \"92ad8452-ea85-436c-824c-e270e48a9c39\") "
Sep 11 12:41:02.527788 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.527714    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-tmp-dir\") pod \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\" (UID: \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\") "
Sep 11 12:41:02.527788 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.527746    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-tls-certs\") pod \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\" (UID: \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\") "
Sep 11 12:41:02.528118 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.527800    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92ad8452-ea85-436c-824c-e270e48a9c39-dshm\") pod \"92ad8452-ea85-436c-824c-e270e48a9c39\" (UID: \"92ad8452-ea85-436c-824c-e270e48a9c39\") "
Sep 11 12:41:02.528118 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.527829    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92ad8452-ea85-436c-824c-e270e48a9c39-home" (OuterVolumeSpecName: "home") pod "92ad8452-ea85-436c-824c-e270e48a9c39" (UID: "92ad8452-ea85-436c-824c-e270e48a9c39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:41:02.528118 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.527860    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-dshm\") pod \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\" (UID: \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\") "
Sep 11 12:41:02.528118 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.527900    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvjk5\" (UniqueName: \"kubernetes.io/projected/92ad8452-ea85-436c-824c-e270e48a9c39-kube-api-access-xvjk5\") pod \"92ad8452-ea85-436c-824c-e270e48a9c39\" (UID: \"92ad8452-ea85-436c-824c-e270e48a9c39\") "
Sep 11 12:41:02.528118 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.527966    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92ad8452-ea85-436c-824c-e270e48a9c39-model-cache\") pod \"92ad8452-ea85-436c-824c-e270e48a9c39\" (UID: \"92ad8452-ea85-436c-824c-e270e48a9c39\") "
Sep 11 12:41:02.528118 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.527995    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-model-cache\") pod \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\" (UID: \"5f6731d4-62c8-4e59-a6be-acc6a9ae6082\") "
Sep 11 12:41:02.528466 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.528336    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92ad8452-ea85-436c-824c-e270e48a9c39-home\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:41:02.528795 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.527965    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-home" (OuterVolumeSpecName: "home") pod "5f6731d4-62c8-4e59-a6be-acc6a9ae6082" (UID: "5f6731d4-62c8-4e59-a6be-acc6a9ae6082"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:41:02.528795 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.527962    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92ad8452-ea85-436c-824c-e270e48a9c39-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "92ad8452-ea85-436c-824c-e270e48a9c39" (UID: "92ad8452-ea85-436c-824c-e270e48a9c39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:41:02.528795 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.528182    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5f6731d4-62c8-4e59-a6be-acc6a9ae6082" (UID: "5f6731d4-62c8-4e59-a6be-acc6a9ae6082"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:41:02.528795 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.528564    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-model-cache" (OuterVolumeSpecName: "model-cache") pod "5f6731d4-62c8-4e59-a6be-acc6a9ae6082" (UID: "5f6731d4-62c8-4e59-a6be-acc6a9ae6082"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:41:02.528795 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.528757    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92ad8452-ea85-436c-824c-e270e48a9c39-model-cache" (OuterVolumeSpecName: "model-cache") pod "92ad8452-ea85-436c-824c-e270e48a9c39" (UID: "92ad8452-ea85-436c-824c-e270e48a9c39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:41:02.530138 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.530112    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-dshm" (OuterVolumeSpecName: "dshm") pod "5f6731d4-62c8-4e59-a6be-acc6a9ae6082" (UID: "5f6731d4-62c8-4e59-a6be-acc6a9ae6082"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:41:02.530407 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.530379    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92ad8452-ea85-436c-824c-e270e48a9c39-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "92ad8452-ea85-436c-824c-e270e48a9c39" (UID: "92ad8452-ea85-436c-824c-e270e48a9c39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:41:02.530407 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.530399    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f6731d4-62c8-4e59-a6be-acc6a9ae6082" (UID: "5f6731d4-62c8-4e59-a6be-acc6a9ae6082"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:41:02.530558 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.530447    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-kube-api-access-mv9mk" (OuterVolumeSpecName: "kube-api-access-mv9mk") pod "5f6731d4-62c8-4e59-a6be-acc6a9ae6082" (UID: "5f6731d4-62c8-4e59-a6be-acc6a9ae6082"). InnerVolumeSpecName "kube-api-access-mv9mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:41:02.530558 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.530481    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92ad8452-ea85-436c-824c-e270e48a9c39-kube-api-access-xvjk5" (OuterVolumeSpecName: "kube-api-access-xvjk5") pod "92ad8452-ea85-436c-824c-e270e48a9c39" (UID: "92ad8452-ea85-436c-824c-e270e48a9c39"). InnerVolumeSpecName "kube-api-access-xvjk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:41:02.530558 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.530544    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92ad8452-ea85-436c-824c-e270e48a9c39-dshm" (OuterVolumeSpecName: "dshm") pod "92ad8452-ea85-436c-824c-e270e48a9c39" (UID: "92ad8452-ea85-436c-824c-e270e48a9c39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:41:02.629613 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.629578    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-dshm\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:41:02.629613 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.629608    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvjk5\" (UniqueName: \"kubernetes.io/projected/92ad8452-ea85-436c-824c-e270e48a9c39-kube-api-access-xvjk5\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:41:02.629613 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.629619    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92ad8452-ea85-436c-824c-e270e48a9c39-model-cache\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:41:02.629842 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.629627    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-model-cache\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:41:02.629842 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.629636    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-home\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:41:02.629842 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.629643    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92ad8452-ea85-436c-824c-e270e48a9c39-tmp-dir\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:41:02.629842 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.629651    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv9mk\" (UniqueName: \"kubernetes.io/projected/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-kube-api-access-mv9mk\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:41:02.629842 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.629663    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92ad8452-ea85-436c-824c-e270e48a9c39-tls-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:41:02.629842 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.629674    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-tmp-dir\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:41:02.629842 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.629682    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f6731d4-62c8-4e59-a6be-acc6a9ae6082-tls-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:41:02.629842 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:02.629689    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92ad8452-ea85-436c-824c-e270e48a9c39-dshm\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:41:03.411783 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:03.411749    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"
Sep 11 12:41:03.412291 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:03.411781    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n" event={"ID":"5f6731d4-62c8-4e59-a6be-acc6a9ae6082","Type":"ContainerDied","Data":"e3a64035c3d35ef4adbd99c3f678ad94eba5400b0fa8b3357501df9de2a505cd"}
Sep 11 12:41:03.412291 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:03.411834    2561 scope.go:117] "RemoveContainer" containerID="efbcc68072ab74d2e00b15a8e42ccf0bc013341d468966807100f1b2a4e0cb1c"
Sep 11 12:41:03.413296 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:03.413262    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm" event={"ID":"92ad8452-ea85-436c-824c-e270e48a9c39","Type":"ContainerDied","Data":"6f3d1c158b9ba4abd494029a63c8de5ea4f80cd9505bb1926b15ca9101f8d966"}
Sep 11 12:41:03.413411 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:03.413296    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"
Sep 11 12:41:03.420289 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:03.420271    2561 scope.go:117] "RemoveContainer" containerID="772daab48494e55059cfc087aa2b56e480442f9c762313c9b95d10d8e87257ce"
Sep 11 12:41:03.434222 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:03.434184    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"]
Sep 11 12:41:03.437518 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:03.437498    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/simulator-w4-cb20c8-kserve-6f4fdcc565-kwp4n"]
Sep 11 12:41:03.447420 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:03.447400    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"]
Sep 11 12:41:03.453115 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:03.453087    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/distinct-2-w4-cb20c8-kserve-7956fb5fd8-kchtm"]
Sep 11 12:41:04.931847 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:04.931803    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f6731d4-62c8-4e59-a6be-acc6a9ae6082" path="/var/lib/kubelet/pods/5f6731d4-62c8-4e59-a6be-acc6a9ae6082/volumes"
Sep 11 12:41:04.932407 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:04.932387    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92ad8452-ea85-436c-824c-e270e48a9c39" path="/var/lib/kubelet/pods/92ad8452-ea85-436c-824c-e270e48a9c39/volumes"
Sep 11 12:41:08.884543 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:08.884509    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7603-29151b"]
Sep 11 12:41:08.884987 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:08.884834    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f6731d4-62c8-4e59-a6be-acc6a9ae6082" containerName="main"
Sep 11 12:41:08.884987 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:08.884845    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f6731d4-62c8-4e59-a6be-acc6a9ae6082" containerName="main"
Sep 11 12:41:08.884987 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:08.884853    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07822040-fed3-4d3e-a34e-e43a30bbf94b" containerName="revoke-keys"
Sep 11 12:41:08.884987 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:08.884858    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="07822040-fed3-4d3e-a34e-e43a30bbf94b" containerName="revoke-keys"
Sep 11 12:41:08.884987 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:08.884869    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d2a4d3c-95d8-4dec-b576-9ef468ffcf42" containerName="payload-pre-processing"
Sep 11 12:41:08.884987 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:08.884875    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d2a4d3c-95d8-4dec-b576-9ef468ffcf42" containerName="payload-pre-processing"
Sep 11 12:41:08.884987 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:08.884883    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f49fb2d-fa85-4cba-a555-6e455299cb06" containerName="payload-processing"
Sep 11 12:41:08.884987 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:08.884889    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f49fb2d-fa85-4cba-a555-6e455299cb06" containerName="payload-processing"
Sep 11 12:41:08.884987 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:08.884896    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92ad8452-ea85-436c-824c-e270e48a9c39" containerName="main"
Sep 11 12:41:08.884987 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:08.884901    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="92ad8452-ea85-436c-824c-e270e48a9c39" containerName="main"
Sep 11 12:41:08.884987 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:08.884953    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d2a4d3c-95d8-4dec-b576-9ef468ffcf42" containerName="payload-pre-processing"
Sep 11 12:41:08.884987 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:08.884960    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f49fb2d-fa85-4cba-a555-6e455299cb06" containerName="payload-processing"
Sep 11 12:41:08.884987 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:08.884966    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f6731d4-62c8-4e59-a6be-acc6a9ae6082" containerName="main"
Sep 11 12:41:08.884987 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:08.884971    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="07822040-fed3-4d3e-a34e-e43a30bbf94b" containerName="revoke-keys"
Sep 11 12:41:08.884987 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:08.884978    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="92ad8452-ea85-436c-824c-e270e48a9c39" containerName="main"
Sep 11 12:41:08.889246 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:08.889229    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-29151b"
Sep 11 12:41:08.891494 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:08.891476    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h59p7\""
Sep 11 12:41:08.894383 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:08.894359    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7603-29151b"]
Sep 11 12:41:08.980220 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:08.980177    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z75dl\" (UniqueName: \"kubernetes.io/projected/374a3380-5fb6-45ca-a66a-f1170360e0ad-kube-api-access-z75dl\") pod \"test-curl-7603-29151b\" (UID: \"374a3380-5fb6-45ca-a66a-f1170360e0ad\") " pod="opendatahub/test-curl-7603-29151b"
Sep 11 12:41:09.080775 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:09.080745    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z75dl\" (UniqueName: \"kubernetes.io/projected/374a3380-5fb6-45ca-a66a-f1170360e0ad-kube-api-access-z75dl\") pod \"test-curl-7603-29151b\" (UID: \"374a3380-5fb6-45ca-a66a-f1170360e0ad\") " pod="opendatahub/test-curl-7603-29151b"
Sep 11 12:41:09.088937 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:09.088915    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z75dl\" (UniqueName: \"kubernetes.io/projected/374a3380-5fb6-45ca-a66a-f1170360e0ad-kube-api-access-z75dl\") pod \"test-curl-7603-29151b\" (UID: \"374a3380-5fb6-45ca-a66a-f1170360e0ad\") " pod="opendatahub/test-curl-7603-29151b"
Sep 11 12:41:09.199759 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:09.199684    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-29151b"
Sep 11 12:41:09.472221 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:09.472180    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7603-29151b"]
Sep 11 12:41:09.473963 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:41:09.473940    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod374a3380_5fb6_45ca_a66a_f1170360e0ad.slice/crio-b3fbbe11efbed2a68a7ecc735d269b3facde0f343f950a468f42cf02341ce034 WatchSource:0}: Error finding container b3fbbe11efbed2a68a7ecc735d269b3facde0f343f950a468f42cf02341ce034: Status 404 returned error can't find the container with id b3fbbe11efbed2a68a7ecc735d269b3facde0f343f950a468f42cf02341ce034
Sep 11 12:41:10.456562 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:10.456528    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7603-29151b" event={"ID":"374a3380-5fb6-45ca-a66a-f1170360e0ad","Type":"ContainerStarted","Data":"635ca2a8fd7ee6d5ecfc3651f4f34edb2f00478fa70329d9aef4137ea17d17c2"}
Sep 11 12:41:10.456562 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:10.456569    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7603-29151b" event={"ID":"374a3380-5fb6-45ca-a66a-f1170360e0ad","Type":"ContainerStarted","Data":"b3fbbe11efbed2a68a7ecc735d269b3facde0f343f950a468f42cf02341ce034"}
Sep 11 12:41:10.473159 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:10.473116    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7603-29151b" podStartSLOduration=2.473100997 podStartE2EDuration="2.473100997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:41:10.47161719 +0000 UTC m=+1602.034396950" watchObservedRunningTime="2026-09-11 12:41:10.473100997 +0000 UTC m=+1602.035880759"
Sep 11 12:41:11.084109 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:11.084072    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7603-29151b"]
Sep 11 12:41:11.089106 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:11.089076    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7603-29151b"]
Sep 11 12:41:11.461876 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:41:11.461847    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7603-29151b" container="test-curl-7603-29151b"
Sep 11 12:41:12.466553 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:12.466511    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7603-29151b" podUID="374a3380-5fb6-45ca-a66a-f1170360e0ad" containerName="test-curl-7603-29151b" containerID="cri-o://635ca2a8fd7ee6d5ecfc3651f4f34edb2f00478fa70329d9aef4137ea17d17c2" gracePeriod=2
Sep 11 12:41:12.469019 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:41:12.468989    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7603-29151b\" is forbidden: User \"system:node:ip-10-0-131-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-172.ec2.internal' and this object" podUID="374a3380-5fb6-45ca-a66a-f1170360e0ad" pod="opendatahub/test-curl-7603-29151b"
Sep 11 12:41:12.924772 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:12.924734    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:41:14.519444 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:14.519410    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"]
Sep 11 12:41:14.625693 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:14.625671    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-29151b"
Sep 11 12:41:14.730464 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:14.730441    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z75dl\" (UniqueName: \"kubernetes.io/projected/374a3380-5fb6-45ca-a66a-f1170360e0ad-kube-api-access-z75dl\") pod \"374a3380-5fb6-45ca-a66a-f1170360e0ad\" (UID: \"374a3380-5fb6-45ca-a66a-f1170360e0ad\") "
Sep 11 12:41:14.732393 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:14.732368    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/374a3380-5fb6-45ca-a66a-f1170360e0ad-kube-api-access-z75dl" (OuterVolumeSpecName: "kube-api-access-z75dl") pod "374a3380-5fb6-45ca-a66a-f1170360e0ad" (UID: "374a3380-5fb6-45ca-a66a-f1170360e0ad"). InnerVolumeSpecName "kube-api-access-z75dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:41:14.831201 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:14.831167    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z75dl\" (UniqueName: \"kubernetes.io/projected/374a3380-5fb6-45ca-a66a-f1170360e0ad-kube-api-access-z75dl\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:41:14.932297 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:14.932266    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="374a3380-5fb6-45ca-a66a-f1170360e0ad" path="/var/lib/kubelet/pods/374a3380-5fb6-45ca-a66a-f1170360e0ad/volumes"
Sep 11 12:41:15.481773 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:15.481738    2561 generic.go:356] "Generic (PLEG): container finished" podID="374a3380-5fb6-45ca-a66a-f1170360e0ad" containerID="635ca2a8fd7ee6d5ecfc3651f4f34edb2f00478fa70329d9aef4137ea17d17c2" exitCode=137
Sep 11 12:41:15.481948 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:15.481776    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-29151b"
Sep 11 12:41:15.481948 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:15.481842    2561 scope.go:117] "RemoveContainer" containerID="635ca2a8fd7ee6d5ecfc3651f4f34edb2f00478fa70329d9aef4137ea17d17c2"
Sep 11 12:41:15.490030 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:15.490013    2561 scope.go:117] "RemoveContainer" containerID="635ca2a8fd7ee6d5ecfc3651f4f34edb2f00478fa70329d9aef4137ea17d17c2"
Sep 11 12:41:15.490344 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:41:15.490324    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"635ca2a8fd7ee6d5ecfc3651f4f34edb2f00478fa70329d9aef4137ea17d17c2\": container with ID starting with 635ca2a8fd7ee6d5ecfc3651f4f34edb2f00478fa70329d9aef4137ea17d17c2 not found: ID does not exist" containerID="635ca2a8fd7ee6d5ecfc3651f4f34edb2f00478fa70329d9aef4137ea17d17c2"
Sep 11 12:41:15.490439 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:15.490353    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"635ca2a8fd7ee6d5ecfc3651f4f34edb2f00478fa70329d9aef4137ea17d17c2"} err="failed to get container status \"635ca2a8fd7ee6d5ecfc3651f4f34edb2f00478fa70329d9aef4137ea17d17c2\": rpc error: code = NotFound desc = could not find container \"635ca2a8fd7ee6d5ecfc3651f4f34edb2f00478fa70329d9aef4137ea17d17c2\": container with ID starting with 635ca2a8fd7ee6d5ecfc3651f4f34edb2f00478fa70329d9aef4137ea17d17c2 not found: ID does not exist"
Sep 11 12:41:23.126002 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:23.125923    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:41:26.698184 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.698148    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"]
Sep 11 12:41:26.698636 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.698497    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="374a3380-5fb6-45ca-a66a-f1170360e0ad" containerName="test-curl-7603-29151b"
Sep 11 12:41:26.698636 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.698508    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="374a3380-5fb6-45ca-a66a-f1170360e0ad" containerName="test-curl-7603-29151b"
Sep 11 12:41:26.698636 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.698566    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="374a3380-5fb6-45ca-a66a-f1170360e0ad" containerName="test-curl-7603-29151b"
Sep 11 12:41:26.702631 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.702613    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:41:26.705254 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.705236    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-aaa824\"/\"kube-root-ca.crt\""
Sep 11 12:41:26.706621 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.706599    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-aaa824\"/\"default-dockercfg-679f9\""
Sep 11 12:41:26.706731 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.706647    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-aaa824\"/\"sub-test-model-aaa824-kserve-self-signed-certs\""
Sep 11 12:41:26.706731 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.706680    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-aaa824\"/\"openshift-service-ca.crt\""
Sep 11 12:41:26.710813 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.710791    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"]
Sep 11 12:41:26.833936 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.833905    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87d58028-aeb9-4f94-8622-89373d568415-model-cache\") pod \"sub-test-model-aaa824-kserve-785d88874f-6w2zt\" (UID: \"87d58028-aeb9-4f94-8622-89373d568415\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:41:26.834069 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.833938    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87d58028-aeb9-4f94-8622-89373d568415-home\") pod \"sub-test-model-aaa824-kserve-785d88874f-6w2zt\" (UID: \"87d58028-aeb9-4f94-8622-89373d568415\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:41:26.834069 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.833963    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87d58028-aeb9-4f94-8622-89373d568415-tmp-dir\") pod \"sub-test-model-aaa824-kserve-785d88874f-6w2zt\" (UID: \"87d58028-aeb9-4f94-8622-89373d568415\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:41:26.834069 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.834036    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87d58028-aeb9-4f94-8622-89373d568415-tls-certs\") pod \"sub-test-model-aaa824-kserve-785d88874f-6w2zt\" (UID: \"87d58028-aeb9-4f94-8622-89373d568415\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:41:26.834182 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.834084    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmnmx\" (UniqueName: \"kubernetes.io/projected/87d58028-aeb9-4f94-8622-89373d568415-kube-api-access-pmnmx\") pod \"sub-test-model-aaa824-kserve-785d88874f-6w2zt\" (UID: \"87d58028-aeb9-4f94-8622-89373d568415\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:41:26.834182 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.834115    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87d58028-aeb9-4f94-8622-89373d568415-dshm\") pod \"sub-test-model-aaa824-kserve-785d88874f-6w2zt\" (UID: \"87d58028-aeb9-4f94-8622-89373d568415\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:41:26.935432 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.935401    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87d58028-aeb9-4f94-8622-89373d568415-model-cache\") pod \"sub-test-model-aaa824-kserve-785d88874f-6w2zt\" (UID: \"87d58028-aeb9-4f94-8622-89373d568415\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:41:26.935432 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.935436    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87d58028-aeb9-4f94-8622-89373d568415-home\") pod \"sub-test-model-aaa824-kserve-785d88874f-6w2zt\" (UID: \"87d58028-aeb9-4f94-8622-89373d568415\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:41:26.935652 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.935463    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87d58028-aeb9-4f94-8622-89373d568415-tmp-dir\") pod \"sub-test-model-aaa824-kserve-785d88874f-6w2zt\" (UID: \"87d58028-aeb9-4f94-8622-89373d568415\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:41:26.935652 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.935505    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87d58028-aeb9-4f94-8622-89373d568415-tls-certs\") pod \"sub-test-model-aaa824-kserve-785d88874f-6w2zt\" (UID: \"87d58028-aeb9-4f94-8622-89373d568415\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:41:26.935652 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.935533    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmnmx\" (UniqueName: \"kubernetes.io/projected/87d58028-aeb9-4f94-8622-89373d568415-kube-api-access-pmnmx\") pod \"sub-test-model-aaa824-kserve-785d88874f-6w2zt\" (UID: \"87d58028-aeb9-4f94-8622-89373d568415\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:41:26.935652 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.935574    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87d58028-aeb9-4f94-8622-89373d568415-dshm\") pod \"sub-test-model-aaa824-kserve-785d88874f-6w2zt\" (UID: \"87d58028-aeb9-4f94-8622-89373d568415\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:41:26.935845 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.935822    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87d58028-aeb9-4f94-8622-89373d568415-model-cache\") pod \"sub-test-model-aaa824-kserve-785d88874f-6w2zt\" (UID: \"87d58028-aeb9-4f94-8622-89373d568415\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:41:26.935900 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.935865    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87d58028-aeb9-4f94-8622-89373d568415-tmp-dir\") pod \"sub-test-model-aaa824-kserve-785d88874f-6w2zt\" (UID: \"87d58028-aeb9-4f94-8622-89373d568415\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:41:26.935956 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.935891    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87d58028-aeb9-4f94-8622-89373d568415-home\") pod \"sub-test-model-aaa824-kserve-785d88874f-6w2zt\" (UID: \"87d58028-aeb9-4f94-8622-89373d568415\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:41:26.937769 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.937748    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87d58028-aeb9-4f94-8622-89373d568415-dshm\") pod \"sub-test-model-aaa824-kserve-785d88874f-6w2zt\" (UID: \"87d58028-aeb9-4f94-8622-89373d568415\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:41:26.938026 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.938010    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87d58028-aeb9-4f94-8622-89373d568415-tls-certs\") pod \"sub-test-model-aaa824-kserve-785d88874f-6w2zt\" (UID: \"87d58028-aeb9-4f94-8622-89373d568415\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:41:26.944108 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:26.944087    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmnmx\" (UniqueName: \"kubernetes.io/projected/87d58028-aeb9-4f94-8622-89373d568415-kube-api-access-pmnmx\") pod \"sub-test-model-aaa824-kserve-785d88874f-6w2zt\" (UID: \"87d58028-aeb9-4f94-8622-89373d568415\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:41:27.013099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:27.013071    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:41:27.284971 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:27.284948    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"]
Sep 11 12:41:27.286936 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:41:27.286908    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87d58028_aeb9_4f94_8622_89373d568415.slice/crio-5f505885e3ec82b9dfa28ae2858ec32d0e3a2e2f42cc6ddff37294d1892e9299 WatchSource:0}: Error finding container 5f505885e3ec82b9dfa28ae2858ec32d0e3a2e2f42cc6ddff37294d1892e9299: Status 404 returned error can't find the container with id 5f505885e3ec82b9dfa28ae2858ec32d0e3a2e2f42cc6ddff37294d1892e9299
Sep 11 12:41:27.544923 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:27.544823    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt" event={"ID":"87d58028-aeb9-4f94-8622-89373d568415","Type":"ContainerStarted","Data":"c504d22720f18598ef78abeac87018f53f05304126cf038f242d0cd4fe8ef118"}
Sep 11 12:41:27.544923 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:27.544877    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt" event={"ID":"87d58028-aeb9-4f94-8622-89373d568415","Type":"ContainerStarted","Data":"5f505885e3ec82b9dfa28ae2858ec32d0e3a2e2f42cc6ddff37294d1892e9299"}
Sep 11 12:41:27.562653 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:27.562604    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt" podStartSLOduration=1.5625904099999999 podStartE2EDuration="1.56259041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:41:27.561009722 +0000 UTC m=+1619.123789508" watchObservedRunningTime="2026-09-11 12:41:27.56259041 +0000 UTC m=+1619.125370173"
Sep 11 12:41:29.539197 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:29.539156    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22" podUID="15013cc9-0933-40e0-958e-bc31edd1f3bf" containerName="main" containerID="cri-o://f650ee7f12e6961e993b27e3d8892c030171e160a6a1c9c03ddde810a1604417" gracePeriod=45
Sep 11 12:41:29.797063 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:29.797007    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:41:29.858501 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:29.858468    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15013cc9-0933-40e0-958e-bc31edd1f3bf-tmp-dir\") pod \"15013cc9-0933-40e0-958e-bc31edd1f3bf\" (UID: \"15013cc9-0933-40e0-958e-bc31edd1f3bf\") "
Sep 11 12:41:29.858647 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:29.858566    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15013cc9-0933-40e0-958e-bc31edd1f3bf-home\") pod \"15013cc9-0933-40e0-958e-bc31edd1f3bf\" (UID: \"15013cc9-0933-40e0-958e-bc31edd1f3bf\") "
Sep 11 12:41:29.858647 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:29.858588    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15013cc9-0933-40e0-958e-bc31edd1f3bf-dshm\") pod \"15013cc9-0933-40e0-958e-bc31edd1f3bf\" (UID: \"15013cc9-0933-40e0-958e-bc31edd1f3bf\") "
Sep 11 12:41:29.858647 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:29.858613    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klhc2\" (UniqueName: \"kubernetes.io/projected/15013cc9-0933-40e0-958e-bc31edd1f3bf-kube-api-access-klhc2\") pod \"15013cc9-0933-40e0-958e-bc31edd1f3bf\" (UID: \"15013cc9-0933-40e0-958e-bc31edd1f3bf\") "
Sep 11 12:41:29.858647 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:29.858630    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15013cc9-0933-40e0-958e-bc31edd1f3bf-tls-certs\") pod \"15013cc9-0933-40e0-958e-bc31edd1f3bf\" (UID: \"15013cc9-0933-40e0-958e-bc31edd1f3bf\") "
Sep 11 12:41:29.858859 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:29.858667    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15013cc9-0933-40e0-958e-bc31edd1f3bf-model-cache\") pod \"15013cc9-0933-40e0-958e-bc31edd1f3bf\" (UID: \"15013cc9-0933-40e0-958e-bc31edd1f3bf\") "
Sep 11 12:41:29.858859 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:29.858763    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15013cc9-0933-40e0-958e-bc31edd1f3bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "15013cc9-0933-40e0-958e-bc31edd1f3bf" (UID: "15013cc9-0933-40e0-958e-bc31edd1f3bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:41:29.858976 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:29.858868    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15013cc9-0933-40e0-958e-bc31edd1f3bf-tmp-dir\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:41:29.858976 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:29.858938    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15013cc9-0933-40e0-958e-bc31edd1f3bf-home" (OuterVolumeSpecName: "home") pod "15013cc9-0933-40e0-958e-bc31edd1f3bf" (UID: "15013cc9-0933-40e0-958e-bc31edd1f3bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:41:29.859047 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:29.858966    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15013cc9-0933-40e0-958e-bc31edd1f3bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "15013cc9-0933-40e0-958e-bc31edd1f3bf" (UID: "15013cc9-0933-40e0-958e-bc31edd1f3bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:41:29.860758 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:29.860731    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15013cc9-0933-40e0-958e-bc31edd1f3bf-dshm" (OuterVolumeSpecName: "dshm") pod "15013cc9-0933-40e0-958e-bc31edd1f3bf" (UID: "15013cc9-0933-40e0-958e-bc31edd1f3bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:41:29.860880 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:29.860762    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15013cc9-0933-40e0-958e-bc31edd1f3bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "15013cc9-0933-40e0-958e-bc31edd1f3bf" (UID: "15013cc9-0933-40e0-958e-bc31edd1f3bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:41:29.860880 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:29.860808    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15013cc9-0933-40e0-958e-bc31edd1f3bf-kube-api-access-klhc2" (OuterVolumeSpecName: "kube-api-access-klhc2") pod "15013cc9-0933-40e0-958e-bc31edd1f3bf" (UID: "15013cc9-0933-40e0-958e-bc31edd1f3bf"). InnerVolumeSpecName "kube-api-access-klhc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:41:29.959577 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:29.959536    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15013cc9-0933-40e0-958e-bc31edd1f3bf-home\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:41:29.959577 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:29.959570    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15013cc9-0933-40e0-958e-bc31edd1f3bf-dshm\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:41:29.959577 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:29.959579    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klhc2\" (UniqueName: \"kubernetes.io/projected/15013cc9-0933-40e0-958e-bc31edd1f3bf-kube-api-access-klhc2\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:41:29.959577 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:29.959591    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15013cc9-0933-40e0-958e-bc31edd1f3bf-tls-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:41:29.959827 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:29.959600    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15013cc9-0933-40e0-958e-bc31edd1f3bf-model-cache\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:41:30.562653 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:30.562616    2561 generic.go:356] "Generic (PLEG): container finished" podID="15013cc9-0933-40e0-958e-bc31edd1f3bf" containerID="f650ee7f12e6961e993b27e3d8892c030171e160a6a1c9c03ddde810a1604417" exitCode=0
Sep 11 12:41:30.562653 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:30.562656    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22" event={"ID":"15013cc9-0933-40e0-958e-bc31edd1f3bf","Type":"ContainerDied","Data":"f650ee7f12e6961e993b27e3d8892c030171e160a6a1c9c03ddde810a1604417"}
Sep 11 12:41:30.563121 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:30.562684    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22" event={"ID":"15013cc9-0933-40e0-958e-bc31edd1f3bf","Type":"ContainerDied","Data":"7f67139942fbe623dbb06ef42abab0fcbfabb0ea7c6963738694561fe11b9d33"}
Sep 11 12:41:30.563121 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:30.562657    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"
Sep 11 12:41:30.563121 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:30.562700    2561 scope.go:117] "RemoveContainer" containerID="f650ee7f12e6961e993b27e3d8892c030171e160a6a1c9c03ddde810a1604417"
Sep 11 12:41:30.571598 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:30.571582    2561 scope.go:117] "RemoveContainer" containerID="f650ee7f12e6961e993b27e3d8892c030171e160a6a1c9c03ddde810a1604417"
Sep 11 12:41:30.571869 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:41:30.571849    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f650ee7f12e6961e993b27e3d8892c030171e160a6a1c9c03ddde810a1604417\": container with ID starting with f650ee7f12e6961e993b27e3d8892c030171e160a6a1c9c03ddde810a1604417 not found: ID does not exist" containerID="f650ee7f12e6961e993b27e3d8892c030171e160a6a1c9c03ddde810a1604417"
Sep 11 12:41:30.571929 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:30.571876    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f650ee7f12e6961e993b27e3d8892c030171e160a6a1c9c03ddde810a1604417"} err="failed to get container status \"f650ee7f12e6961e993b27e3d8892c030171e160a6a1c9c03ddde810a1604417\": rpc error: code = NotFound desc = could not find container \"f650ee7f12e6961e993b27e3d8892c030171e160a6a1c9c03ddde810a1604417\": container with ID starting with f650ee7f12e6961e993b27e3d8892c030171e160a6a1c9c03ddde810a1604417 not found: ID does not exist"
Sep 11 12:41:30.584184 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:30.584159    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"]
Sep 11 12:41:30.588163 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:30.588142    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4502d/auth-test-model-f4502d-kserve-7944f5f479-zfv22"]
Sep 11 12:41:30.932939 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:30.932846    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15013cc9-0933-40e0-958e-bc31edd1f3bf" path="/var/lib/kubelet/pods/15013cc9-0933-40e0-958e-bc31edd1f3bf/volumes"
Sep 11 12:41:31.537847 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:31.537809    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:41:37.013223 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:37.013173    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:41:37.013610 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:37.013256    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:41:37.025799 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:37.025775    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:41:37.608949 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:37.608921    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:41:47.379635 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.379600    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"]
Sep 11 12:41:47.380007 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.379990    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15013cc9-0933-40e0-958e-bc31edd1f3bf" containerName="main"
Sep 11 12:41:47.380052 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.380011    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="15013cc9-0933-40e0-958e-bc31edd1f3bf" containerName="main"
Sep 11 12:41:47.380122 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.380110    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="15013cc9-0933-40e0-958e-bc31edd1f3bf" containerName="main"
Sep 11 12:41:47.384498 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.384477    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:41:47.386878 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.386855    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4502d\"/\"openshift-service-ca.crt\""
Sep 11 12:41:47.387812 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.387792    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4502d\"/\"sub-test-model-f4502d-kserve-self-signed-certs\""
Sep 11 12:41:47.387894 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.387796    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4502d\"/\"kube-root-ca.crt\""
Sep 11 12:41:47.388163 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.388142    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4502d\"/\"default-dockercfg-p8khr\""
Sep 11 12:41:47.393902 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.393877    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"]
Sep 11 12:41:47.515014 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.514986    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkmg5\" (UniqueName: \"kubernetes.io/projected/a5957702-ca34-4600-a927-1eec3fd02a32-kube-api-access-rkmg5\") pod \"sub-test-model-f4502d-kserve-574b9884fc-58qbd\" (UID: \"a5957702-ca34-4600-a927-1eec3fd02a32\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:41:47.515148 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.515027    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5957702-ca34-4600-a927-1eec3fd02a32-tls-certs\") pod \"sub-test-model-f4502d-kserve-574b9884fc-58qbd\" (UID: \"a5957702-ca34-4600-a927-1eec3fd02a32\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:41:47.515148 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.515050    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5957702-ca34-4600-a927-1eec3fd02a32-home\") pod \"sub-test-model-f4502d-kserve-574b9884fc-58qbd\" (UID: \"a5957702-ca34-4600-a927-1eec3fd02a32\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:41:47.515320 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.515138    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5957702-ca34-4600-a927-1eec3fd02a32-tmp-dir\") pod \"sub-test-model-f4502d-kserve-574b9884fc-58qbd\" (UID: \"a5957702-ca34-4600-a927-1eec3fd02a32\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:41:47.515320 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.515178    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5957702-ca34-4600-a927-1eec3fd02a32-dshm\") pod \"sub-test-model-f4502d-kserve-574b9884fc-58qbd\" (UID: \"a5957702-ca34-4600-a927-1eec3fd02a32\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:41:47.515320 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.515266    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5957702-ca34-4600-a927-1eec3fd02a32-model-cache\") pod \"sub-test-model-f4502d-kserve-574b9884fc-58qbd\" (UID: \"a5957702-ca34-4600-a927-1eec3fd02a32\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:41:47.615780 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.615756    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkmg5\" (UniqueName: \"kubernetes.io/projected/a5957702-ca34-4600-a927-1eec3fd02a32-kube-api-access-rkmg5\") pod \"sub-test-model-f4502d-kserve-574b9884fc-58qbd\" (UID: \"a5957702-ca34-4600-a927-1eec3fd02a32\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:41:47.615914 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.615796    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5957702-ca34-4600-a927-1eec3fd02a32-tls-certs\") pod \"sub-test-model-f4502d-kserve-574b9884fc-58qbd\" (UID: \"a5957702-ca34-4600-a927-1eec3fd02a32\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:41:47.615914 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.615822    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5957702-ca34-4600-a927-1eec3fd02a32-home\") pod \"sub-test-model-f4502d-kserve-574b9884fc-58qbd\" (UID: \"a5957702-ca34-4600-a927-1eec3fd02a32\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:41:47.615914 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.615856    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5957702-ca34-4600-a927-1eec3fd02a32-tmp-dir\") pod \"sub-test-model-f4502d-kserve-574b9884fc-58qbd\" (UID: \"a5957702-ca34-4600-a927-1eec3fd02a32\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:41:47.615914 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.615883    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5957702-ca34-4600-a927-1eec3fd02a32-dshm\") pod \"sub-test-model-f4502d-kserve-574b9884fc-58qbd\" (UID: \"a5957702-ca34-4600-a927-1eec3fd02a32\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:41:47.616064 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.615938    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5957702-ca34-4600-a927-1eec3fd02a32-model-cache\") pod \"sub-test-model-f4502d-kserve-574b9884fc-58qbd\" (UID: \"a5957702-ca34-4600-a927-1eec3fd02a32\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:41:47.616332 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.616309    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5957702-ca34-4600-a927-1eec3fd02a32-model-cache\") pod \"sub-test-model-f4502d-kserve-574b9884fc-58qbd\" (UID: \"a5957702-ca34-4600-a927-1eec3fd02a32\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:41:47.616409 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.616366    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5957702-ca34-4600-a927-1eec3fd02a32-tmp-dir\") pod \"sub-test-model-f4502d-kserve-574b9884fc-58qbd\" (UID: \"a5957702-ca34-4600-a927-1eec3fd02a32\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:41:47.616409 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.616400    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5957702-ca34-4600-a927-1eec3fd02a32-home\") pod \"sub-test-model-f4502d-kserve-574b9884fc-58qbd\" (UID: \"a5957702-ca34-4600-a927-1eec3fd02a32\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:41:47.618094 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.618074    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5957702-ca34-4600-a927-1eec3fd02a32-dshm\") pod \"sub-test-model-f4502d-kserve-574b9884fc-58qbd\" (UID: \"a5957702-ca34-4600-a927-1eec3fd02a32\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:41:47.618331 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.618312    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5957702-ca34-4600-a927-1eec3fd02a32-tls-certs\") pod \"sub-test-model-f4502d-kserve-574b9884fc-58qbd\" (UID: \"a5957702-ca34-4600-a927-1eec3fd02a32\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:41:47.624216 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.624174    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkmg5\" (UniqueName: \"kubernetes.io/projected/a5957702-ca34-4600-a927-1eec3fd02a32-kube-api-access-rkmg5\") pod \"sub-test-model-f4502d-kserve-574b9884fc-58qbd\" (UID: \"a5957702-ca34-4600-a927-1eec3fd02a32\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:41:47.694735 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.694679    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:41:47.970530 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:47.970506    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"]
Sep 11 12:41:47.971914 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:41:47.971882    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5957702_ca34_4600_a927_1eec3fd02a32.slice/crio-98301c407b146eb8d69e37bc4370621663395ad1526f19a5480d39ff3148614f WatchSource:0}: Error finding container 98301c407b146eb8d69e37bc4370621663395ad1526f19a5480d39ff3148614f: Status 404 returned error can't find the container with id 98301c407b146eb8d69e37bc4370621663395ad1526f19a5480d39ff3148614f
Sep 11 12:41:48.650250 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:48.650217    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd" event={"ID":"a5957702-ca34-4600-a927-1eec3fd02a32","Type":"ContainerStarted","Data":"13628348f64c09b19fcc2487b79c884d5a61cc0549ba2a9f362661dbf7a4535f"}
Sep 11 12:41:48.650608 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:48.650259    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd" event={"ID":"a5957702-ca34-4600-a927-1eec3fd02a32","Type":"ContainerStarted","Data":"98301c407b146eb8d69e37bc4370621663395ad1526f19a5480d39ff3148614f"}
Sep 11 12:41:48.669849 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:48.669792    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd" podStartSLOduration=1.66977713 podStartE2EDuration="1.66977713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:41:48.669355936 +0000 UTC m=+1640.232135698" watchObservedRunningTime="2026-09-11 12:41:48.66977713 +0000 UTC m=+1640.232556892"
Sep 11 12:41:50.539479 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:50.539438    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:41:57.695362 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:57.695334    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:41:57.695854 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:57.695572    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:41:57.707573 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:57.707550    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:41:58.704767 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:41:58.704740    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:42:12.229438 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:12.229396    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:42:24.234528 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:24.234491    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:42:26.828924 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:26.828891    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7603-6c7a63"]
Sep 11 12:42:26.832414 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:26.832398    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-6c7a63"
Sep 11 12:42:26.834714 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:26.834692    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h59p7\""
Sep 11 12:42:26.838046 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:26.838021    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7603-6c7a63"]
Sep 11 12:42:26.951275 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:26.951228    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgqg6\" (UniqueName: \"kubernetes.io/projected/193b22c7-5aef-4e95-9c1b-a31f6aed8fa3-kube-api-access-rgqg6\") pod \"test-curl-7603-6c7a63\" (UID: \"193b22c7-5aef-4e95-9c1b-a31f6aed8fa3\") " pod="opendatahub/test-curl-7603-6c7a63"
Sep 11 12:42:27.052716 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:27.052684    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgqg6\" (UniqueName: \"kubernetes.io/projected/193b22c7-5aef-4e95-9c1b-a31f6aed8fa3-kube-api-access-rgqg6\") pod \"test-curl-7603-6c7a63\" (UID: \"193b22c7-5aef-4e95-9c1b-a31f6aed8fa3\") " pod="opendatahub/test-curl-7603-6c7a63"
Sep 11 12:42:27.061891 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:27.061862    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgqg6\" (UniqueName: \"kubernetes.io/projected/193b22c7-5aef-4e95-9c1b-a31f6aed8fa3-kube-api-access-rgqg6\") pod \"test-curl-7603-6c7a63\" (UID: \"193b22c7-5aef-4e95-9c1b-a31f6aed8fa3\") " pod="opendatahub/test-curl-7603-6c7a63"
Sep 11 12:42:27.143922 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:27.143852    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-6c7a63"
Sep 11 12:42:27.413864 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:27.413836    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7603-6c7a63"]
Sep 11 12:42:27.415981 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:42:27.415953    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod193b22c7_5aef_4e95_9c1b_a31f6aed8fa3.slice/crio-a3f6cd1c323f62f2370f9698c2532e9cb567637f783b47a4332864c7b8568928 WatchSource:0}: Error finding container a3f6cd1c323f62f2370f9698c2532e9cb567637f783b47a4332864c7b8568928: Status 404 returned error can't find the container with id a3f6cd1c323f62f2370f9698c2532e9cb567637f783b47a4332864c7b8568928
Sep 11 12:42:27.830238 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:27.830203    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7603-6c7a63" event={"ID":"193b22c7-5aef-4e95-9c1b-a31f6aed8fa3","Type":"ContainerStarted","Data":"9eba6f5c055df12ab407612011e0e932413523304e579abca4ad246846537e3a"}
Sep 11 12:42:27.830599 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:27.830246    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7603-6c7a63" event={"ID":"193b22c7-5aef-4e95-9c1b-a31f6aed8fa3","Type":"ContainerStarted","Data":"a3f6cd1c323f62f2370f9698c2532e9cb567637f783b47a4332864c7b8568928"}
Sep 11 12:42:27.848959 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:27.848917    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7603-6c7a63" podStartSLOduration=1.848901984 podStartE2EDuration="1.848901984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:42:27.847142967 +0000 UTC m=+1679.409922731" watchObservedRunningTime="2026-09-11 12:42:27.848901984 +0000 UTC m=+1679.411681747"
Sep 11 12:42:28.428996 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:28.428960    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7603-6c7a63"]
Sep 11 12:42:28.436565 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:28.436534    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7603-6c7a63"]
Sep 11 12:42:28.557807 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:28.557768    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7603-68f185"]
Sep 11 12:42:28.558112 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:28.558099    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="193b22c7-5aef-4e95-9c1b-a31f6aed8fa3" containerName="test-curl-7603-6c7a63"
Sep 11 12:42:28.558156 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:28.558112    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="193b22c7-5aef-4e95-9c1b-a31f6aed8fa3" containerName="test-curl-7603-6c7a63"
Sep 11 12:42:28.558221 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:28.558205    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="193b22c7-5aef-4e95-9c1b-a31f6aed8fa3" containerName="test-curl-7603-6c7a63"
Sep 11 12:42:28.561224 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:28.561181    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-68f185"
Sep 11 12:42:28.569012 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:28.568985    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7603-68f185"]
Sep 11 12:42:28.669272 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:28.669236    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njnkv\" (UniqueName: \"kubernetes.io/projected/7432adfc-3d15-4fd8-ac7b-097a2860323b-kube-api-access-njnkv\") pod \"test-curl-7603-68f185\" (UID: \"7432adfc-3d15-4fd8-ac7b-097a2860323b\") " pod="opendatahub/test-curl-7603-68f185"
Sep 11 12:42:28.770085 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:28.770050    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njnkv\" (UniqueName: \"kubernetes.io/projected/7432adfc-3d15-4fd8-ac7b-097a2860323b-kube-api-access-njnkv\") pod \"test-curl-7603-68f185\" (UID: \"7432adfc-3d15-4fd8-ac7b-097a2860323b\") " pod="opendatahub/test-curl-7603-68f185"
Sep 11 12:42:28.778160 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:28.778134    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njnkv\" (UniqueName: \"kubernetes.io/projected/7432adfc-3d15-4fd8-ac7b-097a2860323b-kube-api-access-njnkv\") pod \"test-curl-7603-68f185\" (UID: \"7432adfc-3d15-4fd8-ac7b-097a2860323b\") " pod="opendatahub/test-curl-7603-68f185"
Sep 11 12:42:28.835074 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:42:28.834985    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7603-6c7a63" container="test-curl-7603-6c7a63"
Sep 11 12:42:28.871016 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:28.870981    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-68f185"
Sep 11 12:42:29.139589 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:29.139563    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7603-68f185"]
Sep 11 12:42:29.141311 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:42:29.141271    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7432adfc_3d15_4fd8_ac7b_097a2860323b.slice/crio-0adb86d00f55282713d003a77329be4811fb1a59fe1f26657aa00e3029301823 WatchSource:0}: Error finding container 0adb86d00f55282713d003a77329be4811fb1a59fe1f26657aa00e3029301823: Status 404 returned error can't find the container with id 0adb86d00f55282713d003a77329be4811fb1a59fe1f26657aa00e3029301823
Sep 11 12:42:29.841036 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:29.840997    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7603-68f185" event={"ID":"7432adfc-3d15-4fd8-ac7b-097a2860323b","Type":"ContainerStarted","Data":"25fa2b31535b431765fbfe20121471de03cd16471d84e4a67cbdffdf9d7bf184"}
Sep 11 12:42:29.841036 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:29.841022    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7603-6c7a63" podUID="193b22c7-5aef-4e95-9c1b-a31f6aed8fa3" containerName="test-curl-7603-6c7a63" containerID="cri-o://9eba6f5c055df12ab407612011e0e932413523304e579abca4ad246846537e3a" gracePeriod=2
Sep 11 12:42:29.841036 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:29.841043    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7603-68f185" event={"ID":"7432adfc-3d15-4fd8-ac7b-097a2860323b","Type":"ContainerStarted","Data":"0adb86d00f55282713d003a77329be4811fb1a59fe1f26657aa00e3029301823"}
Sep 11 12:42:29.857129 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:29.857081    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7603-68f185" podStartSLOduration=1.8570642739999998 podStartE2EDuration="1.857064274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:42:29.856910614 +0000 UTC m=+1681.419690378" watchObservedRunningTime="2026-09-11 12:42:29.857064274 +0000 UTC m=+1681.419844038"
Sep 11 12:42:30.426857 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:30.426819    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7603-68f185"]
Sep 11 12:42:30.433560 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:30.433534    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7603-68f185"]
Sep 11 12:42:30.845659 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:42:30.845627    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7603-68f185" container="test-curl-7603-68f185"
Sep 11 12:42:31.851039 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:31.850999    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7603-68f185" podUID="7432adfc-3d15-4fd8-ac7b-097a2860323b" containerName="test-curl-7603-68f185" containerID="cri-o://25fa2b31535b431765fbfe20121471de03cd16471d84e4a67cbdffdf9d7bf184" gracePeriod=2
Sep 11 12:42:31.980993 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:31.980967    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-6c7a63"
Sep 11 12:42:32.100361 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:32.100293    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgqg6\" (UniqueName: \"kubernetes.io/projected/193b22c7-5aef-4e95-9c1b-a31f6aed8fa3-kube-api-access-rgqg6\") pod \"193b22c7-5aef-4e95-9c1b-a31f6aed8fa3\" (UID: \"193b22c7-5aef-4e95-9c1b-a31f6aed8fa3\") "
Sep 11 12:42:32.102375 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:32.102351    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193b22c7-5aef-4e95-9c1b-a31f6aed8fa3-kube-api-access-rgqg6" (OuterVolumeSpecName: "kube-api-access-rgqg6") pod "193b22c7-5aef-4e95-9c1b-a31f6aed8fa3" (UID: "193b22c7-5aef-4e95-9c1b-a31f6aed8fa3"). InnerVolumeSpecName "kube-api-access-rgqg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:42:32.201601 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:32.201575    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgqg6\" (UniqueName: \"kubernetes.io/projected/193b22c7-5aef-4e95-9c1b-a31f6aed8fa3-kube-api-access-rgqg6\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:42:32.472061 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:32.471980    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"]
Sep 11 12:42:32.856677 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:32.856646    2561 generic.go:356] "Generic (PLEG): container finished" podID="193b22c7-5aef-4e95-9c1b-a31f6aed8fa3" containerID="9eba6f5c055df12ab407612011e0e932413523304e579abca4ad246846537e3a" exitCode=137
Sep 11 12:42:32.857064 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:32.856688    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-6c7a63"
Sep 11 12:42:32.857064 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:32.856732    2561 scope.go:117] "RemoveContainer" containerID="9eba6f5c055df12ab407612011e0e932413523304e579abca4ad246846537e3a"
Sep 11 12:42:32.866108 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:32.866090    2561 scope.go:117] "RemoveContainer" containerID="9eba6f5c055df12ab407612011e0e932413523304e579abca4ad246846537e3a"
Sep 11 12:42:32.866476 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:42:32.866455    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eba6f5c055df12ab407612011e0e932413523304e579abca4ad246846537e3a\": container with ID starting with 9eba6f5c055df12ab407612011e0e932413523304e579abca4ad246846537e3a not found: ID does not exist" containerID="9eba6f5c055df12ab407612011e0e932413523304e579abca4ad246846537e3a"
Sep 11 12:42:32.866546 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:32.866490    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eba6f5c055df12ab407612011e0e932413523304e579abca4ad246846537e3a"} err="failed to get container status \"9eba6f5c055df12ab407612011e0e932413523304e579abca4ad246846537e3a\": rpc error: code = NotFound desc = could not find container \"9eba6f5c055df12ab407612011e0e932413523304e579abca4ad246846537e3a\": container with ID starting with 9eba6f5c055df12ab407612011e0e932413523304e579abca4ad246846537e3a not found: ID does not exist"
Sep 11 12:42:32.867427 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:42:32.867408    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7603-6c7a63\" is forbidden: User \"system:node:ip-10-0-131-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-172.ec2.internal' and this object" podUID="193b22c7-5aef-4e95-9c1b-a31f6aed8fa3" pod="opendatahub/test-curl-7603-6c7a63"
Sep 11 12:42:32.932076 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:32.932048    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="193b22c7-5aef-4e95-9c1b-a31f6aed8fa3" path="/var/lib/kubelet/pods/193b22c7-5aef-4e95-9c1b-a31f6aed8fa3/volumes"
Sep 11 12:42:33.385815 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:33.385783    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"]
Sep 11 12:42:34.017387 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:34.017362    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-68f185"
Sep 11 12:42:34.120290 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:34.120221    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njnkv\" (UniqueName: \"kubernetes.io/projected/7432adfc-3d15-4fd8-ac7b-097a2860323b-kube-api-access-njnkv\") pod \"7432adfc-3d15-4fd8-ac7b-097a2860323b\" (UID: \"7432adfc-3d15-4fd8-ac7b-097a2860323b\") "
Sep 11 12:42:34.122146 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:34.122121    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7432adfc-3d15-4fd8-ac7b-097a2860323b-kube-api-access-njnkv" (OuterVolumeSpecName: "kube-api-access-njnkv") pod "7432adfc-3d15-4fd8-ac7b-097a2860323b" (UID: "7432adfc-3d15-4fd8-ac7b-097a2860323b"). InnerVolumeSpecName "kube-api-access-njnkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:42:34.221455 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:34.221416    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njnkv\" (UniqueName: \"kubernetes.io/projected/7432adfc-3d15-4fd8-ac7b-097a2860323b-kube-api-access-njnkv\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:42:34.527307 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:34.527274    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:42:34.872850 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:34.872762    2561 generic.go:356] "Generic (PLEG): container finished" podID="7432adfc-3d15-4fd8-ac7b-097a2860323b" containerID="25fa2b31535b431765fbfe20121471de03cd16471d84e4a67cbdffdf9d7bf184" exitCode=137
Sep 11 12:42:34.872850 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:34.872803    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-68f185"
Sep 11 12:42:34.873056 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:34.872866    2561 scope.go:117] "RemoveContainer" containerID="25fa2b31535b431765fbfe20121471de03cd16471d84e4a67cbdffdf9d7bf184"
Sep 11 12:42:34.881235 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:34.881218    2561 scope.go:117] "RemoveContainer" containerID="25fa2b31535b431765fbfe20121471de03cd16471d84e4a67cbdffdf9d7bf184"
Sep 11 12:42:34.881505 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:42:34.881486    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25fa2b31535b431765fbfe20121471de03cd16471d84e4a67cbdffdf9d7bf184\": container with ID starting with 25fa2b31535b431765fbfe20121471de03cd16471d84e4a67cbdffdf9d7bf184 not found: ID does not exist" containerID="25fa2b31535b431765fbfe20121471de03cd16471d84e4a67cbdffdf9d7bf184"
Sep 11 12:42:34.881576 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:34.881513    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25fa2b31535b431765fbfe20121471de03cd16471d84e4a67cbdffdf9d7bf184"} err="failed to get container status \"25fa2b31535b431765fbfe20121471de03cd16471d84e4a67cbdffdf9d7bf184\": rpc error: code = NotFound desc = could not find container \"25fa2b31535b431765fbfe20121471de03cd16471d84e4a67cbdffdf9d7bf184\": container with ID starting with 25fa2b31535b431765fbfe20121471de03cd16471d84e4a67cbdffdf9d7bf184 not found: ID does not exist"
Sep 11 12:42:34.883123 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:42:34.883096    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7603-68f185\" is forbidden: User \"system:node:ip-10-0-131-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-172.ec2.internal' and this object" podUID="7432adfc-3d15-4fd8-ac7b-097a2860323b" pod="opendatahub/test-curl-7603-68f185"
Sep 11 12:42:34.932569 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:34.932540    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7432adfc-3d15-4fd8-ac7b-097a2860323b" path="/var/lib/kubelet/pods/7432adfc-3d15-4fd8-ac7b-097a2860323b/volumes"
Sep 11 12:42:38.434972 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.434938    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"]
Sep 11 12:42:38.435343 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.435302    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7432adfc-3d15-4fd8-ac7b-097a2860323b" containerName="test-curl-7603-68f185"
Sep 11 12:42:38.435343 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.435315    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7432adfc-3d15-4fd8-ac7b-097a2860323b" containerName="test-curl-7603-68f185"
Sep 11 12:42:38.435416 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.435376    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7432adfc-3d15-4fd8-ac7b-097a2860323b" containerName="test-curl-7603-68f185"
Sep 11 12:42:38.440302 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.440286    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:42:38.444674 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.444658    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-aaa824\"/\"rate-test-model-aaa824-kserve-self-signed-certs\""
Sep 11 12:42:38.451525 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.451501    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"]
Sep 11 12:42:38.561065 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.561028    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-dshm\") pod \"rate-test-model-aaa824-kserve-84cf66bbbc-xczh2\" (UID: \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:42:38.561262 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.561070    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-home\") pod \"rate-test-model-aaa824-kserve-84cf66bbbc-xczh2\" (UID: \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:42:38.561262 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.561097    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fftmw\" (UniqueName: \"kubernetes.io/projected/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-kube-api-access-fftmw\") pod \"rate-test-model-aaa824-kserve-84cf66bbbc-xczh2\" (UID: \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:42:38.561262 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.561165    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-tls-certs\") pod \"rate-test-model-aaa824-kserve-84cf66bbbc-xczh2\" (UID: \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:42:38.561262 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.561236    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-tmp-dir\") pod \"rate-test-model-aaa824-kserve-84cf66bbbc-xczh2\" (UID: \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:42:38.561262 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.561264    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-model-cache\") pod \"rate-test-model-aaa824-kserve-84cf66bbbc-xczh2\" (UID: \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:42:38.662482 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.662442    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-dshm\") pod \"rate-test-model-aaa824-kserve-84cf66bbbc-xczh2\" (UID: \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:42:38.662653 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.662490    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-home\") pod \"rate-test-model-aaa824-kserve-84cf66bbbc-xczh2\" (UID: \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:42:38.662653 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.662616    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fftmw\" (UniqueName: \"kubernetes.io/projected/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-kube-api-access-fftmw\") pod \"rate-test-model-aaa824-kserve-84cf66bbbc-xczh2\" (UID: \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:42:38.662766 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.662655    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-tls-certs\") pod \"rate-test-model-aaa824-kserve-84cf66bbbc-xczh2\" (UID: \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:42:38.662766 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.662706    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-tmp-dir\") pod \"rate-test-model-aaa824-kserve-84cf66bbbc-xczh2\" (UID: \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:42:38.662766 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.662745    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-model-cache\") pod \"rate-test-model-aaa824-kserve-84cf66bbbc-xczh2\" (UID: \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:42:38.662947 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.662927    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-home\") pod \"rate-test-model-aaa824-kserve-84cf66bbbc-xczh2\" (UID: \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:42:38.663069 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.663045    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-tmp-dir\") pod \"rate-test-model-aaa824-kserve-84cf66bbbc-xczh2\" (UID: \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:42:38.663142 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.663118    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-model-cache\") pod \"rate-test-model-aaa824-kserve-84cf66bbbc-xczh2\" (UID: \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:42:38.664749 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.664730    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-dshm\") pod \"rate-test-model-aaa824-kserve-84cf66bbbc-xczh2\" (UID: \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:42:38.665090 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.665076    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-tls-certs\") pod \"rate-test-model-aaa824-kserve-84cf66bbbc-xczh2\" (UID: \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:42:38.670326 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.670306    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fftmw\" (UniqueName: \"kubernetes.io/projected/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-kube-api-access-fftmw\") pod \"rate-test-model-aaa824-kserve-84cf66bbbc-xczh2\" (UID: \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:42:38.748781 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:38.748743    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:42:39.019693 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:39.019627    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"]
Sep 11 12:42:39.022116 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:42:39.022090    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod077caf26_2a66_4ea1_aa0c_3ad9ad516e04.slice/crio-43cd0d912721aafa770a6b3fee88fb315c26acf107435777673f4cf0ae0db650 WatchSource:0}: Error finding container 43cd0d912721aafa770a6b3fee88fb315c26acf107435777673f4cf0ae0db650: Status 404 returned error can't find the container with id 43cd0d912721aafa770a6b3fee88fb315c26acf107435777673f4cf0ae0db650
Sep 11 12:42:39.898375 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:39.898336    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2" event={"ID":"077caf26-2a66-4ea1-aa0c-3ad9ad516e04","Type":"ContainerStarted","Data":"70cc63372ed61f056dc833325cca9af2145f512294aafd876baccdb517ef6669"}
Sep 11 12:42:39.898375 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:39.898381    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2" event={"ID":"077caf26-2a66-4ea1-aa0c-3ad9ad516e04","Type":"ContainerStarted","Data":"43cd0d912721aafa770a6b3fee88fb315c26acf107435777673f4cf0ae0db650"}
Sep 11 12:42:39.916855 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:39.916809    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2" podStartSLOduration=1.9167908329999999 podStartE2EDuration="1.916790833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:42:39.916112384 +0000 UTC m=+1691.478892147" watchObservedRunningTime="2026-09-11 12:42:39.916790833 +0000 UTC m=+1691.479570596"
Sep 11 12:42:47.491699 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.491655    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt" podUID="87d58028-aeb9-4f94-8622-89373d568415" containerName="main" containerID="cri-o://c504d22720f18598ef78abeac87018f53f05304126cf038f242d0cd4fe8ef118" gracePeriod=45
Sep 11 12:42:47.598626 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.598576    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt" podUID="87d58028-aeb9-4f94-8622-89373d568415" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 11 12:42:47.747861 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.747807    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:42:47.825839 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.825814    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:42:47.846170 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.846137    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87d58028-aeb9-4f94-8622-89373d568415-tmp-dir\") pod \"87d58028-aeb9-4f94-8622-89373d568415\" (UID: \"87d58028-aeb9-4f94-8622-89373d568415\") "
Sep 11 12:42:47.846268 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.846179    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87d58028-aeb9-4f94-8622-89373d568415-tls-certs\") pod \"87d58028-aeb9-4f94-8622-89373d568415\" (UID: \"87d58028-aeb9-4f94-8622-89373d568415\") "
Sep 11 12:42:47.846268 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.846228    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87d58028-aeb9-4f94-8622-89373d568415-dshm\") pod \"87d58028-aeb9-4f94-8622-89373d568415\" (UID: \"87d58028-aeb9-4f94-8622-89373d568415\") "
Sep 11 12:42:47.846268 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.846263    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmnmx\" (UniqueName: \"kubernetes.io/projected/87d58028-aeb9-4f94-8622-89373d568415-kube-api-access-pmnmx\") pod \"87d58028-aeb9-4f94-8622-89373d568415\" (UID: \"87d58028-aeb9-4f94-8622-89373d568415\") "
Sep 11 12:42:47.846404 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.846298    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87d58028-aeb9-4f94-8622-89373d568415-home\") pod \"87d58028-aeb9-4f94-8622-89373d568415\" (UID: \"87d58028-aeb9-4f94-8622-89373d568415\") "
Sep 11 12:42:47.846404 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.846331    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87d58028-aeb9-4f94-8622-89373d568415-model-cache\") pod \"87d58028-aeb9-4f94-8622-89373d568415\" (UID: \"87d58028-aeb9-4f94-8622-89373d568415\") "
Sep 11 12:42:47.846404 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.846386    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d58028-aeb9-4f94-8622-89373d568415-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "87d58028-aeb9-4f94-8622-89373d568415" (UID: "87d58028-aeb9-4f94-8622-89373d568415"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:42:47.846606 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.846563    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d58028-aeb9-4f94-8622-89373d568415-home" (OuterVolumeSpecName: "home") pod "87d58028-aeb9-4f94-8622-89373d568415" (UID: "87d58028-aeb9-4f94-8622-89373d568415"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:42:47.846691 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.846604    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d58028-aeb9-4f94-8622-89373d568415-model-cache" (OuterVolumeSpecName: "model-cache") pod "87d58028-aeb9-4f94-8622-89373d568415" (UID: "87d58028-aeb9-4f94-8622-89373d568415"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:42:47.846691 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.846616    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87d58028-aeb9-4f94-8622-89373d568415-tmp-dir\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:42:47.848571 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.848549    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87d58028-aeb9-4f94-8622-89373d568415-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "87d58028-aeb9-4f94-8622-89373d568415" (UID: "87d58028-aeb9-4f94-8622-89373d568415"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:42:47.848571 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.848561    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d58028-aeb9-4f94-8622-89373d568415-dshm" (OuterVolumeSpecName: "dshm") pod "87d58028-aeb9-4f94-8622-89373d568415" (UID: "87d58028-aeb9-4f94-8622-89373d568415"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:42:47.848745 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.848723    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d58028-aeb9-4f94-8622-89373d568415-kube-api-access-pmnmx" (OuterVolumeSpecName: "kube-api-access-pmnmx") pod "87d58028-aeb9-4f94-8622-89373d568415" (UID: "87d58028-aeb9-4f94-8622-89373d568415"). InnerVolumeSpecName "kube-api-access-pmnmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:42:47.938713 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.938681    2561 generic.go:356] "Generic (PLEG): container finished" podID="87d58028-aeb9-4f94-8622-89373d568415" containerID="c504d22720f18598ef78abeac87018f53f05304126cf038f242d0cd4fe8ef118" exitCode=0
Sep 11 12:42:47.938873 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.938723    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"
Sep 11 12:42:47.938873 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.938770    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt" event={"ID":"87d58028-aeb9-4f94-8622-89373d568415","Type":"ContainerDied","Data":"c504d22720f18598ef78abeac87018f53f05304126cf038f242d0cd4fe8ef118"}
Sep 11 12:42:47.938873 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.938809    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt" event={"ID":"87d58028-aeb9-4f94-8622-89373d568415","Type":"ContainerDied","Data":"5f505885e3ec82b9dfa28ae2858ec32d0e3a2e2f42cc6ddff37294d1892e9299"}
Sep 11 12:42:47.938873 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.938827    2561 scope.go:117] "RemoveContainer" containerID="c504d22720f18598ef78abeac87018f53f05304126cf038f242d0cd4fe8ef118"
Sep 11 12:42:47.947382 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.947359    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87d58028-aeb9-4f94-8622-89373d568415-tls-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:42:47.947382 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.947383    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87d58028-aeb9-4f94-8622-89373d568415-dshm\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:42:47.947550 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.947393    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmnmx\" (UniqueName: \"kubernetes.io/projected/87d58028-aeb9-4f94-8622-89373d568415-kube-api-access-pmnmx\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:42:47.947550 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.947402    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87d58028-aeb9-4f94-8622-89373d568415-home\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:42:47.947550 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.947411    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87d58028-aeb9-4f94-8622-89373d568415-model-cache\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:42:47.948043 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.948021    2561 scope.go:117] "RemoveContainer" containerID="c504d22720f18598ef78abeac87018f53f05304126cf038f242d0cd4fe8ef118"
Sep 11 12:42:47.948348 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:42:47.948329    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c504d22720f18598ef78abeac87018f53f05304126cf038f242d0cd4fe8ef118\": container with ID starting with c504d22720f18598ef78abeac87018f53f05304126cf038f242d0cd4fe8ef118 not found: ID does not exist" containerID="c504d22720f18598ef78abeac87018f53f05304126cf038f242d0cd4fe8ef118"
Sep 11 12:42:47.948409 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.948358    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c504d22720f18598ef78abeac87018f53f05304126cf038f242d0cd4fe8ef118"} err="failed to get container status \"c504d22720f18598ef78abeac87018f53f05304126cf038f242d0cd4fe8ef118\": rpc error: code = NotFound desc = could not find container \"c504d22720f18598ef78abeac87018f53f05304126cf038f242d0cd4fe8ef118\": container with ID starting with c504d22720f18598ef78abeac87018f53f05304126cf038f242d0cd4fe8ef118 not found: ID does not exist"
Sep 11 12:42:47.959979 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.959955    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"]
Sep 11 12:42:47.963476 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:47.963453    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aaa824/sub-test-model-aaa824-kserve-785d88874f-6w2zt"]
Sep 11 12:42:48.405251 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.405207    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd" podUID="a5957702-ca34-4600-a927-1eec3fd02a32" containerName="main" containerID="cri-o://13628348f64c09b19fcc2487b79c884d5a61cc0549ba2a9f362661dbf7a4535f" gracePeriod=45
Sep 11 12:42:48.653117 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.653092    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:42:48.748991 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.748948    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:42:48.749152 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.749010    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:42:48.755644 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.755619    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5957702-ca34-4600-a927-1eec3fd02a32-home\") pod \"a5957702-ca34-4600-a927-1eec3fd02a32\" (UID: \"a5957702-ca34-4600-a927-1eec3fd02a32\") "
Sep 11 12:42:48.755784 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.755667    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5957702-ca34-4600-a927-1eec3fd02a32-tmp-dir\") pod \"a5957702-ca34-4600-a927-1eec3fd02a32\" (UID: \"a5957702-ca34-4600-a927-1eec3fd02a32\") "
Sep 11 12:42:48.755784 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.755706    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5957702-ca34-4600-a927-1eec3fd02a32-dshm\") pod \"a5957702-ca34-4600-a927-1eec3fd02a32\" (UID: \"a5957702-ca34-4600-a927-1eec3fd02a32\") "
Sep 11 12:42:48.755784 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.755756    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5957702-ca34-4600-a927-1eec3fd02a32-model-cache\") pod \"a5957702-ca34-4600-a927-1eec3fd02a32\" (UID: \"a5957702-ca34-4600-a927-1eec3fd02a32\") "
Sep 11 12:42:48.755944 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.755814    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5957702-ca34-4600-a927-1eec3fd02a32-tls-certs\") pod \"a5957702-ca34-4600-a927-1eec3fd02a32\" (UID: \"a5957702-ca34-4600-a927-1eec3fd02a32\") "
Sep 11 12:42:48.755944 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.755855    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkmg5\" (UniqueName: \"kubernetes.io/projected/a5957702-ca34-4600-a927-1eec3fd02a32-kube-api-access-rkmg5\") pod \"a5957702-ca34-4600-a927-1eec3fd02a32\" (UID: \"a5957702-ca34-4600-a927-1eec3fd02a32\") "
Sep 11 12:42:48.755944 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.755870    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5957702-ca34-4600-a927-1eec3fd02a32-home" (OuterVolumeSpecName: "home") pod "a5957702-ca34-4600-a927-1eec3fd02a32" (UID: "a5957702-ca34-4600-a927-1eec3fd02a32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:42:48.756089 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.755972    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5957702-ca34-4600-a927-1eec3fd02a32-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5957702-ca34-4600-a927-1eec3fd02a32" (UID: "a5957702-ca34-4600-a927-1eec3fd02a32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:42:48.756089 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.756016    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5957702-ca34-4600-a927-1eec3fd02a32-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5957702-ca34-4600-a927-1eec3fd02a32" (UID: "a5957702-ca34-4600-a927-1eec3fd02a32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:42:48.756214 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.756176    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5957702-ca34-4600-a927-1eec3fd02a32-home\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:42:48.756214 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.756209    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5957702-ca34-4600-a927-1eec3fd02a32-tmp-dir\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:42:48.756325 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.756225    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5957702-ca34-4600-a927-1eec3fd02a32-model-cache\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:42:48.757796 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.757778    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5957702-ca34-4600-a927-1eec3fd02a32-dshm" (OuterVolumeSpecName: "dshm") pod "a5957702-ca34-4600-a927-1eec3fd02a32" (UID: "a5957702-ca34-4600-a927-1eec3fd02a32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:42:48.758102 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.758084    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5957702-ca34-4600-a927-1eec3fd02a32-kube-api-access-rkmg5" (OuterVolumeSpecName: "kube-api-access-rkmg5") pod "a5957702-ca34-4600-a927-1eec3fd02a32" (UID: "a5957702-ca34-4600-a927-1eec3fd02a32"). InnerVolumeSpecName "kube-api-access-rkmg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:42:48.758154 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.758146    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5957702-ca34-4600-a927-1eec3fd02a32-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5957702-ca34-4600-a927-1eec3fd02a32" (UID: "a5957702-ca34-4600-a927-1eec3fd02a32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:42:48.762203 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.762171    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:42:48.856755 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.856729    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5957702-ca34-4600-a927-1eec3fd02a32-dshm\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:42:48.856755 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.856752    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5957702-ca34-4600-a927-1eec3fd02a32-tls-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:42:48.856909 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.856763    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkmg5\" (UniqueName: \"kubernetes.io/projected/a5957702-ca34-4600-a927-1eec3fd02a32-kube-api-access-rkmg5\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:42:48.932999 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.932960    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87d58028-aeb9-4f94-8622-89373d568415" path="/var/lib/kubelet/pods/87d58028-aeb9-4f94-8622-89373d568415/volumes"
Sep 11 12:42:48.944014 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.943987    2561 generic.go:356] "Generic (PLEG): container finished" podID="a5957702-ca34-4600-a927-1eec3fd02a32" containerID="13628348f64c09b19fcc2487b79c884d5a61cc0549ba2a9f362661dbf7a4535f" exitCode=0
Sep 11 12:42:48.944100 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.944019    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"
Sep 11 12:42:48.944100 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.944073    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd" event={"ID":"a5957702-ca34-4600-a927-1eec3fd02a32","Type":"ContainerDied","Data":"13628348f64c09b19fcc2487b79c884d5a61cc0549ba2a9f362661dbf7a4535f"}
Sep 11 12:42:48.944210 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.944107    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd" event={"ID":"a5957702-ca34-4600-a927-1eec3fd02a32","Type":"ContainerDied","Data":"98301c407b146eb8d69e37bc4370621663395ad1526f19a5480d39ff3148614f"}
Sep 11 12:42:48.944210 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.944124    2561 scope.go:117] "RemoveContainer" containerID="13628348f64c09b19fcc2487b79c884d5a61cc0549ba2a9f362661dbf7a4535f"
Sep 11 12:42:48.952387 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.952364    2561 scope.go:117] "RemoveContainer" containerID="13628348f64c09b19fcc2487b79c884d5a61cc0549ba2a9f362661dbf7a4535f"
Sep 11 12:42:48.952640 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:42:48.952619    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13628348f64c09b19fcc2487b79c884d5a61cc0549ba2a9f362661dbf7a4535f\": container with ID starting with 13628348f64c09b19fcc2487b79c884d5a61cc0549ba2a9f362661dbf7a4535f not found: ID does not exist" containerID="13628348f64c09b19fcc2487b79c884d5a61cc0549ba2a9f362661dbf7a4535f"
Sep 11 12:42:48.952700 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.952646    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13628348f64c09b19fcc2487b79c884d5a61cc0549ba2a9f362661dbf7a4535f"} err="failed to get container status \"13628348f64c09b19fcc2487b79c884d5a61cc0549ba2a9f362661dbf7a4535f\": rpc error: code = NotFound desc = could not find container \"13628348f64c09b19fcc2487b79c884d5a61cc0549ba2a9f362661dbf7a4535f\": container with ID starting with 13628348f64c09b19fcc2487b79c884d5a61cc0549ba2a9f362661dbf7a4535f not found: ID does not exist"
Sep 11 12:42:48.957605 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.957551    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:42:48.959665 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.959645    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"]
Sep 11 12:42:48.965131 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:48.965112    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4502d/sub-test-model-f4502d-kserve-574b9884fc-58qbd"]
Sep 11 12:42:50.931812 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:50.931778    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5957702-ca34-4600-a927-1eec3fd02a32" path="/var/lib/kubelet/pods/a5957702-ca34-4600-a927-1eec3fd02a32/volumes"
Sep 11 12:42:55.301596 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.301557    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"]
Sep 11 12:42:55.301951 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.301884    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87d58028-aeb9-4f94-8622-89373d568415" containerName="main"
Sep 11 12:42:55.301951 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.301896    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="87d58028-aeb9-4f94-8622-89373d568415" containerName="main"
Sep 11 12:42:55.301951 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.301916    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5957702-ca34-4600-a927-1eec3fd02a32" containerName="main"
Sep 11 12:42:55.301951 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.301921    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5957702-ca34-4600-a927-1eec3fd02a32" containerName="main"
Sep 11 12:42:55.302088 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.301982    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5957702-ca34-4600-a927-1eec3fd02a32" containerName="main"
Sep 11 12:42:55.302088 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.301990    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="87d58028-aeb9-4f94-8622-89373d568415" containerName="main"
Sep 11 12:42:55.306364 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.306346    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:42:55.310081 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.310056    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4502d\"/\"kube-root-ca.crt\""
Sep 11 12:42:55.310242 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.310086    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4502d\"/\"default-dockercfg-p8khr\""
Sep 11 12:42:55.310242 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.310068    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4502d\"/\"openshift-service-ca.crt\""
Sep 11 12:42:55.310242 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.310068    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f4502d\"/\"rate-test-model-f4502d-kserve-self-signed-certs\""
Sep 11 12:42:55.317012 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.316987    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"]
Sep 11 12:42:55.406004 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.405977    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-model-cache\") pod \"rate-test-model-f4502d-kserve-6584d986ff-vs644\" (UID: \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:42:55.406119 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.406012    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb844\" (UniqueName: \"kubernetes.io/projected/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-kube-api-access-kb844\") pod \"rate-test-model-f4502d-kserve-6584d986ff-vs644\" (UID: \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:42:55.406119 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.406040    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-dshm\") pod \"rate-test-model-f4502d-kserve-6584d986ff-vs644\" (UID: \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:42:55.406211 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.406141    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-tls-certs\") pod \"rate-test-model-f4502d-kserve-6584d986ff-vs644\" (UID: \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:42:55.406211 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.406169    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-home\") pod \"rate-test-model-f4502d-kserve-6584d986ff-vs644\" (UID: \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:42:55.406291 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.406249    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-tmp-dir\") pod \"rate-test-model-f4502d-kserve-6584d986ff-vs644\" (UID: \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:42:55.506881 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.506841    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-tmp-dir\") pod \"rate-test-model-f4502d-kserve-6584d986ff-vs644\" (UID: \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:42:55.507044 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.506890    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-model-cache\") pod \"rate-test-model-f4502d-kserve-6584d986ff-vs644\" (UID: \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:42:55.507044 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.506926    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb844\" (UniqueName: \"kubernetes.io/projected/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-kube-api-access-kb844\") pod \"rate-test-model-f4502d-kserve-6584d986ff-vs644\" (UID: \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:42:55.507044 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.506967    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-dshm\") pod \"rate-test-model-f4502d-kserve-6584d986ff-vs644\" (UID: \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:42:55.507044 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.507018    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-tls-certs\") pod \"rate-test-model-f4502d-kserve-6584d986ff-vs644\" (UID: \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:42:55.507044 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.507040    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-home\") pod \"rate-test-model-f4502d-kserve-6584d986ff-vs644\" (UID: \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:42:55.507387 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.507282    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-tmp-dir\") pod \"rate-test-model-f4502d-kserve-6584d986ff-vs644\" (UID: \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:42:55.507387 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.507299    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-model-cache\") pod \"rate-test-model-f4502d-kserve-6584d986ff-vs644\" (UID: \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:42:55.507491 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.507429    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-home\") pod \"rate-test-model-f4502d-kserve-6584d986ff-vs644\" (UID: \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:42:55.509222 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.509185    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-dshm\") pod \"rate-test-model-f4502d-kserve-6584d986ff-vs644\" (UID: \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:42:55.509421 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.509404    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-tls-certs\") pod \"rate-test-model-f4502d-kserve-6584d986ff-vs644\" (UID: \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:42:55.515714 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.515695    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb844\" (UniqueName: \"kubernetes.io/projected/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-kube-api-access-kb844\") pod \"rate-test-model-f4502d-kserve-6584d986ff-vs644\" (UID: \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\") " pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:42:55.616536 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.616465    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:42:55.890617 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.890587    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"]
Sep 11 12:42:55.893227 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:42:55.893200    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6c1f073_9ddb_48e8_9a53_0f0f33493c73.slice/crio-b75b1a8a7fc6b9179633fd868554e30b5895337f850419d9990b65600a3d9da1 WatchSource:0}: Error finding container b75b1a8a7fc6b9179633fd868554e30b5895337f850419d9990b65600a3d9da1: Status 404 returned error can't find the container with id b75b1a8a7fc6b9179633fd868554e30b5895337f850419d9990b65600a3d9da1
Sep 11 12:42:55.978551 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.978522    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644" event={"ID":"d6c1f073-9ddb-48e8-9a53-0f0f33493c73","Type":"ContainerStarted","Data":"1ed12939648d544c5cb0ee8521a7f6b446d2847aa21e33ee3925987ee9987bf7"}
Sep 11 12:42:55.978667 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.978568    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644" event={"ID":"d6c1f073-9ddb-48e8-9a53-0f0f33493c73","Type":"ContainerStarted","Data":"b75b1a8a7fc6b9179633fd868554e30b5895337f850419d9990b65600a3d9da1"}
Sep 11 12:42:55.995431 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:55.995389    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644" podStartSLOduration=0.995373314 podStartE2EDuration="995.373314ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:42:55.99445544 +0000 UTC m=+1707.557235205" watchObservedRunningTime="2026-09-11 12:42:55.995373314 +0000 UTC m=+1707.558153077"
Sep 11 12:42:58.835805 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:42:58.835764    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:43:05.617280 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:05.617241    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:43:05.617778 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:05.617307    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:43:05.629970 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:05.629940    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:43:06.036698 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:06.036671    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:43:15.736023 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:15.735986    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:43:24.233229 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:24.233184    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:43:30.090782 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:30.090747    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"]
Sep 11 12:43:31.008662 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:31.008634    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"]
Sep 11 12:43:33.630285 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:33.630251    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:43:35.976652 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:35.976624    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh"]
Sep 11 12:43:35.979943 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:35.979927    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh"
Sep 11 12:43:35.983478 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:35.983451    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-0a1087-serving-cert\""
Sep 11 12:43:35.984169 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:35.984150    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l5tx4\""
Sep 11 12:43:35.984304 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:35.984177    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 12:43:35.984513 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:35.984498    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 12:43:35.984594 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:35.984496    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 12:43:36.012309 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.012283    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh"]
Sep 11 12:43:36.165931 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.165903    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f97f04-3cc3-4fa0-99ce-471ed9a389a2-metrics-certs\") pod \"maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh\" (UID: \"28f97f04-3cc3-4fa0-99ce-471ed9a389a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh"
Sep 11 12:43:36.166090 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.165939    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f97f04-3cc3-4fa0-99ce-471ed9a389a2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh\" (UID: \"28f97f04-3cc3-4fa0-99ce-471ed9a389a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh"
Sep 11 12:43:36.166090 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.165975    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbjbp\" (UniqueName: \"kubernetes.io/projected/28f97f04-3cc3-4fa0-99ce-471ed9a389a2-kube-api-access-mbjbp\") pod \"maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh\" (UID: \"28f97f04-3cc3-4fa0-99ce-471ed9a389a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh"
Sep 11 12:43:36.166090 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.166009    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f97f04-3cc3-4fa0-99ce-471ed9a389a2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh\" (UID: \"28f97f04-3cc3-4fa0-99ce-471ed9a389a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh"
Sep 11 12:43:36.267314 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.267285    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f97f04-3cc3-4fa0-99ce-471ed9a389a2-metrics-certs\") pod \"maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh\" (UID: \"28f97f04-3cc3-4fa0-99ce-471ed9a389a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh"
Sep 11 12:43:36.267480 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.267326    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f97f04-3cc3-4fa0-99ce-471ed9a389a2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh\" (UID: \"28f97f04-3cc3-4fa0-99ce-471ed9a389a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh"
Sep 11 12:43:36.267480 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.267360    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbjbp\" (UniqueName: \"kubernetes.io/projected/28f97f04-3cc3-4fa0-99ce-471ed9a389a2-kube-api-access-mbjbp\") pod \"maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh\" (UID: \"28f97f04-3cc3-4fa0-99ce-471ed9a389a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh"
Sep 11 12:43:36.267480 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.267382    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f97f04-3cc3-4fa0-99ce-471ed9a389a2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh\" (UID: \"28f97f04-3cc3-4fa0-99ce-471ed9a389a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh"
Sep 11 12:43:36.268075 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.268045    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f97f04-3cc3-4fa0-99ce-471ed9a389a2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh\" (UID: \"28f97f04-3cc3-4fa0-99ce-471ed9a389a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh"
Sep 11 12:43:36.269684 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.269665    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f97f04-3cc3-4fa0-99ce-471ed9a389a2-metrics-certs\") pod \"maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh\" (UID: \"28f97f04-3cc3-4fa0-99ce-471ed9a389a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh"
Sep 11 12:43:36.269792 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.269708    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f97f04-3cc3-4fa0-99ce-471ed9a389a2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh\" (UID: \"28f97f04-3cc3-4fa0-99ce-471ed9a389a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh"
Sep 11 12:43:36.275668 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.275647    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbjbp\" (UniqueName: \"kubernetes.io/projected/28f97f04-3cc3-4fa0-99ce-471ed9a389a2-kube-api-access-mbjbp\") pod \"maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh\" (UID: \"28f97f04-3cc3-4fa0-99ce-471ed9a389a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh"
Sep 11 12:43:36.290099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.290075    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh"
Sep 11 12:43:36.562264 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.562233    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh"]
Sep 11 12:43:36.565077 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:43:36.565052    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f97f04_3cc3_4fa0_99ce_471ed9a389a2.slice/crio-024c54b783c19aa7b209b15ac4680e63baa82e5519f303c0643820c2c9105f2e WatchSource:0}: Error finding container 024c54b783c19aa7b209b15ac4680e63baa82e5519f303c0643820c2c9105f2e: Status 404 returned error can't find the container with id 024c54b783c19aa7b209b15ac4680e63baa82e5519f303c0643820c2c9105f2e
Sep 11 12:43:36.566907 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.566886    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 12:43:36.737921 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.737890    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv"]
Sep 11 12:43:36.742325 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.742308    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv"
Sep 11 12:43:36.745124 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.745103    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0a1087\""
Sep 11 12:43:36.745306 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.745286    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0a1087-dockercfg-5cjtw\""
Sep 11 12:43:36.751682 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.751658    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv"]
Sep 11 12:43:36.873670 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.873580    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlwkg\" (UniqueName: \"kubernetes.io/projected/62a6ff6a-3c11-4036-9d57-636a55aa0ca1-kube-api-access-nlwkg\") pod \"payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv\" (UID: \"62a6ff6a-3c11-4036-9d57-636a55aa0ca1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv"
Sep 11 12:43:36.873670 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.873638    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a6ff6a-3c11-4036-9d57-636a55aa0ca1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv\" (UID: \"62a6ff6a-3c11-4036-9d57-636a55aa0ca1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv"
Sep 11 12:43:36.975120 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.975085    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlwkg\" (UniqueName: \"kubernetes.io/projected/62a6ff6a-3c11-4036-9d57-636a55aa0ca1-kube-api-access-nlwkg\") pod \"payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv\" (UID: \"62a6ff6a-3c11-4036-9d57-636a55aa0ca1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv"
Sep 11 12:43:36.975314 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.975138    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a6ff6a-3c11-4036-9d57-636a55aa0ca1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv\" (UID: \"62a6ff6a-3c11-4036-9d57-636a55aa0ca1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv"
Sep 11 12:43:36.975837 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.975814    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a6ff6a-3c11-4036-9d57-636a55aa0ca1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv\" (UID: \"62a6ff6a-3c11-4036-9d57-636a55aa0ca1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv"
Sep 11 12:43:36.983099 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:36.983072    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlwkg\" (UniqueName: \"kubernetes.io/projected/62a6ff6a-3c11-4036-9d57-636a55aa0ca1-kube-api-access-nlwkg\") pod \"payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv\" (UID: \"62a6ff6a-3c11-4036-9d57-636a55aa0ca1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv"
Sep 11 12:43:37.053969 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:37.053949    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv"
Sep 11 12:43:37.171931 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:37.171847    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh" event={"ID":"28f97f04-3cc3-4fa0-99ce-471ed9a389a2","Type":"ContainerStarted","Data":"e253aaf8524bb84969349e766af52e61f86a2f73bbc6f3107ba336d825b96d3e"}
Sep 11 12:43:37.172075 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:37.171933    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh"
Sep 11 12:43:37.172075 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:37.171958    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh" event={"ID":"28f97f04-3cc3-4fa0-99ce-471ed9a389a2","Type":"ContainerStarted","Data":"024c54b783c19aa7b209b15ac4680e63baa82e5519f303c0643820c2c9105f2e"}
Sep 11 12:43:37.191484 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:37.191441    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh" podStartSLOduration=1.73624973 podStartE2EDuration="2.191427952s" totalImagesPullingTime="455.178222ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:43:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:43:37.190203183 +0000 UTC m=+1748.752982938" watchObservedRunningTime="2026-09-11 12:43:37.191427952 +0000 UTC m=+1748.754207714"
Sep 11 12:43:37.328613 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:37.328588    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv"]
Sep 11 12:43:37.330444 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:43:37.330409    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a6ff6a_3c11_4036_9d57_636a55aa0ca1.slice/crio-24402314cfefdeeb87a127abc7f4eb6e54081d552e7d2f7c8589843796d61049 WatchSource:0}: Error finding container 24402314cfefdeeb87a127abc7f4eb6e54081d552e7d2f7c8589843796d61049: Status 404 returned error can't find the container with id 24402314cfefdeeb87a127abc7f4eb6e54081d552e7d2f7c8589843796d61049
Sep 11 12:43:38.178579 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:38.178543    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv" event={"ID":"62a6ff6a-3c11-4036-9d57-636a55aa0ca1","Type":"ContainerStarted","Data":"1b1cf1d126d557ce970b8dcff3468e5563273c8abd825480bb77cb0ff98a8cfa"}
Sep 11 12:43:38.179035 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:38.178594    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv" event={"ID":"62a6ff6a-3c11-4036-9d57-636a55aa0ca1","Type":"ContainerStarted","Data":"24402314cfefdeeb87a127abc7f4eb6e54081d552e7d2f7c8589843796d61049"}
Sep 11 12:43:38.179035 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:38.178831    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv"
Sep 11 12:43:38.198001 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:38.197961    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv" podStartSLOduration=2.197948655 podStartE2EDuration="2.197948655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:43:38.196519418 +0000 UTC m=+1749.759299182" watchObservedRunningTime="2026-09-11 12:43:38.197948655 +0000 UTC m=+1749.760728418"
Sep 11 12:43:43.184320 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:43.184292    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh"
Sep 11 12:43:45.109801 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:45.109757    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2" podUID="077caf26-2a66-4ea1-aa0c-3ad9ad516e04" containerName="main" containerID="cri-o://70cc63372ed61f056dc833325cca9af2145f512294aafd876baccdb517ef6669" gracePeriod=45
Sep 11 12:43:45.356286 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:45.356264    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:43:45.452812 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:45.452734    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fftmw\" (UniqueName: \"kubernetes.io/projected/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-kube-api-access-fftmw\") pod \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\" (UID: \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\") "
Sep 11 12:43:45.452812 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:45.452812    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-dshm\") pod \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\" (UID: \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\") "
Sep 11 12:43:45.453037 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:45.452830    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-tls-certs\") pod \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\" (UID: \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\") "
Sep 11 12:43:45.453037 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:45.452851    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-model-cache\") pod \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\" (UID: \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\") "
Sep 11 12:43:45.453037 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:45.452899    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-home\") pod \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\" (UID: \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\") "
Sep 11 12:43:45.453037 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:45.452924    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-tmp-dir\") pod \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\" (UID: \"077caf26-2a66-4ea1-aa0c-3ad9ad516e04\") "
Sep 11 12:43:45.453274 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:45.453179    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-home" (OuterVolumeSpecName: "home") pod "077caf26-2a66-4ea1-aa0c-3ad9ad516e04" (UID: "077caf26-2a66-4ea1-aa0c-3ad9ad516e04"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:43:45.453274 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:45.453217    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-model-cache" (OuterVolumeSpecName: "model-cache") pod "077caf26-2a66-4ea1-aa0c-3ad9ad516e04" (UID: "077caf26-2a66-4ea1-aa0c-3ad9ad516e04"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:43:45.453274 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:45.453261    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "077caf26-2a66-4ea1-aa0c-3ad9ad516e04" (UID: "077caf26-2a66-4ea1-aa0c-3ad9ad516e04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:43:45.454931 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:45.454902    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-kube-api-access-fftmw" (OuterVolumeSpecName: "kube-api-access-fftmw") pod "077caf26-2a66-4ea1-aa0c-3ad9ad516e04" (UID: "077caf26-2a66-4ea1-aa0c-3ad9ad516e04"). InnerVolumeSpecName "kube-api-access-fftmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:43:45.455367 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:45.455344    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "077caf26-2a66-4ea1-aa0c-3ad9ad516e04" (UID: "077caf26-2a66-4ea1-aa0c-3ad9ad516e04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:43:45.455367 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:45.455355    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-dshm" (OuterVolumeSpecName: "dshm") pod "077caf26-2a66-4ea1-aa0c-3ad9ad516e04" (UID: "077caf26-2a66-4ea1-aa0c-3ad9ad516e04"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:43:45.554021 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:45.553997    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fftmw\" (UniqueName: \"kubernetes.io/projected/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-kube-api-access-fftmw\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:43:45.554021 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:45.554017    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-dshm\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:43:45.554161 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:45.554027    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-tls-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:43:45.554161 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:45.554038    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-model-cache\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:43:45.554161 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:45.554046    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-home\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:43:45.554161 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:45.554053    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/077caf26-2a66-4ea1-aa0c-3ad9ad516e04-tmp-dir\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:43:46.028780 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.028744    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644" podUID="d6c1f073-9ddb-48e8-9a53-0f0f33493c73" containerName="main" containerID="cri-o://1ed12939648d544c5cb0ee8521a7f6b446d2847aa21e33ee3925987ee9987bf7" gracePeriod=45
Sep 11 12:43:46.032610 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.032570    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644" podUID="d6c1f073-9ddb-48e8-9a53-0f0f33493c73" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": EOF"
Sep 11 12:43:46.218485 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.218453    2561 generic.go:356] "Generic (PLEG): container finished" podID="077caf26-2a66-4ea1-aa0c-3ad9ad516e04" containerID="70cc63372ed61f056dc833325cca9af2145f512294aafd876baccdb517ef6669" exitCode=0
Sep 11 12:43:46.218983 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.218516    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"
Sep 11 12:43:46.218983 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.218541    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2" event={"ID":"077caf26-2a66-4ea1-aa0c-3ad9ad516e04","Type":"ContainerDied","Data":"70cc63372ed61f056dc833325cca9af2145f512294aafd876baccdb517ef6669"}
Sep 11 12:43:46.218983 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.218586    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2" event={"ID":"077caf26-2a66-4ea1-aa0c-3ad9ad516e04","Type":"ContainerDied","Data":"43cd0d912721aafa770a6b3fee88fb315c26acf107435777673f4cf0ae0db650"}
Sep 11 12:43:46.218983 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.218608    2561 scope.go:117] "RemoveContainer" containerID="70cc63372ed61f056dc833325cca9af2145f512294aafd876baccdb517ef6669"
Sep 11 12:43:46.220569 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.220548    2561 generic.go:356] "Generic (PLEG): container finished" podID="d6c1f073-9ddb-48e8-9a53-0f0f33493c73" containerID="1ed12939648d544c5cb0ee8521a7f6b446d2847aa21e33ee3925987ee9987bf7" exitCode=0
Sep 11 12:43:46.220661 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.220616    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644" event={"ID":"d6c1f073-9ddb-48e8-9a53-0f0f33493c73","Type":"ContainerDied","Data":"1ed12939648d544c5cb0ee8521a7f6b446d2847aa21e33ee3925987ee9987bf7"}
Sep 11 12:43:46.229908 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.229890    2561 scope.go:117] "RemoveContainer" containerID="70cc63372ed61f056dc833325cca9af2145f512294aafd876baccdb517ef6669"
Sep 11 12:43:46.230216 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:43:46.230181    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70cc63372ed61f056dc833325cca9af2145f512294aafd876baccdb517ef6669\": container with ID starting with 70cc63372ed61f056dc833325cca9af2145f512294aafd876baccdb517ef6669 not found: ID does not exist" containerID="70cc63372ed61f056dc833325cca9af2145f512294aafd876baccdb517ef6669"
Sep 11 12:43:46.230296 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.230222    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70cc63372ed61f056dc833325cca9af2145f512294aafd876baccdb517ef6669"} err="failed to get container status \"70cc63372ed61f056dc833325cca9af2145f512294aafd876baccdb517ef6669\": rpc error: code = NotFound desc = could not find container \"70cc63372ed61f056dc833325cca9af2145f512294aafd876baccdb517ef6669\": container with ID starting with 70cc63372ed61f056dc833325cca9af2145f512294aafd876baccdb517ef6669 not found: ID does not exist"
Sep 11 12:43:46.240913 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.240893    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"]
Sep 11 12:43:46.244363 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.244340    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aaa824/rate-test-model-aaa824-kserve-84cf66bbbc-xczh2"]
Sep 11 12:43:46.275165 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.275147    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:43:46.360680 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.360597    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-model-cache\") pod \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\" (UID: \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\") "
Sep 11 12:43:46.360680 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.360644    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kb844\" (UniqueName: \"kubernetes.io/projected/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-kube-api-access-kb844\") pod \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\" (UID: \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\") "
Sep 11 12:43:46.360680 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.360673    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-home\") pod \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\" (UID: \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\") "
Sep 11 12:43:46.360942 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.360705    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-tmp-dir\") pod \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\" (UID: \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\") "
Sep 11 12:43:46.360942 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.360740    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-tls-certs\") pod \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\" (UID: \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\") "
Sep 11 12:43:46.360942 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.360823    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-dshm\") pod \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\" (UID: \"d6c1f073-9ddb-48e8-9a53-0f0f33493c73\") "
Sep 11 12:43:46.360942 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.360910    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-model-cache" (OuterVolumeSpecName: "model-cache") pod "d6c1f073-9ddb-48e8-9a53-0f0f33493c73" (UID: "d6c1f073-9ddb-48e8-9a53-0f0f33493c73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:43:46.361137 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.360970    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d6c1f073-9ddb-48e8-9a53-0f0f33493c73" (UID: "d6c1f073-9ddb-48e8-9a53-0f0f33493c73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:43:46.361137 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.360984    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-home" (OuterVolumeSpecName: "home") pod "d6c1f073-9ddb-48e8-9a53-0f0f33493c73" (UID: "d6c1f073-9ddb-48e8-9a53-0f0f33493c73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:43:46.361232 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.361218    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-model-cache\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:43:46.361278 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.361235    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-home\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:43:46.361278 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.361249    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-tmp-dir\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:43:46.362912 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.362891    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6c1f073-9ddb-48e8-9a53-0f0f33493c73" (UID: "d6c1f073-9ddb-48e8-9a53-0f0f33493c73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:43:46.362994 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.362965    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-dshm" (OuterVolumeSpecName: "dshm") pod "d6c1f073-9ddb-48e8-9a53-0f0f33493c73" (UID: "d6c1f073-9ddb-48e8-9a53-0f0f33493c73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:43:46.362994 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.362975    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-kube-api-access-kb844" (OuterVolumeSpecName: "kube-api-access-kb844") pod "d6c1f073-9ddb-48e8-9a53-0f0f33493c73" (UID: "d6c1f073-9ddb-48e8-9a53-0f0f33493c73"). InnerVolumeSpecName "kube-api-access-kb844". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:43:46.462553 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.462529    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-tls-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:43:46.462553 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.462550    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-dshm\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:43:46.462698 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.462561    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb844\" (UniqueName: \"kubernetes.io/projected/d6c1f073-9ddb-48e8-9a53-0f0f33493c73-kube-api-access-kb844\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:43:46.734148 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.734113    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:43:46.932441 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:46.932405    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="077caf26-2a66-4ea1-aa0c-3ad9ad516e04" path="/var/lib/kubelet/pods/077caf26-2a66-4ea1-aa0c-3ad9ad516e04/volumes"
Sep 11 12:43:47.228026 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:47.227944    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"
Sep 11 12:43:47.228026 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:47.227988    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644" event={"ID":"d6c1f073-9ddb-48e8-9a53-0f0f33493c73","Type":"ContainerDied","Data":"b75b1a8a7fc6b9179633fd868554e30b5895337f850419d9990b65600a3d9da1"}
Sep 11 12:43:47.228583 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:47.228034    2561 scope.go:117] "RemoveContainer" containerID="1ed12939648d544c5cb0ee8521a7f6b446d2847aa21e33ee3925987ee9987bf7"
Sep 11 12:43:47.244454 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:47.244428    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"]
Sep 11 12:43:47.246645 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:47.246619    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f4502d/rate-test-model-f4502d-kserve-6584d986ff-vs644"]
Sep 11 12:43:48.933834 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:48.933791    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6c1f073-9ddb-48e8-9a53-0f0f33493c73" path="/var/lib/kubelet/pods/d6c1f073-9ddb-48e8-9a53-0f0f33493c73/volumes"
Sep 11 12:43:49.185573 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:43:49.185489    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv"
Sep 11 12:44:05.201223 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:05.201172    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"]
Sep 11 12:44:05.201700 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:05.201532    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6c1f073-9ddb-48e8-9a53-0f0f33493c73" containerName="main"
Sep 11 12:44:05.201700 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:05.201544    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6c1f073-9ddb-48e8-9a53-0f0f33493c73" containerName="main"
Sep 11 12:44:05.201700 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:05.201564    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="077caf26-2a66-4ea1-aa0c-3ad9ad516e04" containerName="main"
Sep 11 12:44:05.201700 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:05.201570    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="077caf26-2a66-4ea1-aa0c-3ad9ad516e04" containerName="main"
Sep 11 12:44:05.201700 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:05.201625    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="077caf26-2a66-4ea1-aa0c-3ad9ad516e04" containerName="main"
Sep 11 12:44:05.201700 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:05.201634    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6c1f073-9ddb-48e8-9a53-0f0f33493c73" containerName="main"
Sep 11 12:44:05.204564 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:05.204539    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"
Sep 11 12:44:05.207022 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:05.207003    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d5ae2a-serving-cert\""
Sep 11 12:44:05.216274 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:05.216253    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"]
Sep 11 12:44:05.322210 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:05.322164    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf5kd\" (UniqueName: \"kubernetes.io/projected/f5128075-e465-439f-ac27-63a19d7f8ec9-kube-api-access-bf5kd\") pod \"maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms\" (UID: \"f5128075-e465-439f-ac27-63a19d7f8ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"
Sep 11 12:44:05.322425 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:05.322236    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5128075-e465-439f-ac27-63a19d7f8ec9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms\" (UID: \"f5128075-e465-439f-ac27-63a19d7f8ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"
Sep 11 12:44:05.322425 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:05.322267    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5128075-e465-439f-ac27-63a19d7f8ec9-metrics-certs\") pod \"maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms\" (UID: \"f5128075-e465-439f-ac27-63a19d7f8ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"
Sep 11 12:44:05.322425 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:05.322305    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5128075-e465-439f-ac27-63a19d7f8ec9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms\" (UID: \"f5128075-e465-439f-ac27-63a19d7f8ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"
Sep 11 12:44:05.423127 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:05.423088    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5128075-e465-439f-ac27-63a19d7f8ec9-metrics-certs\") pod \"maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms\" (UID: \"f5128075-e465-439f-ac27-63a19d7f8ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"
Sep 11 12:44:05.423298 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:05.423147    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5128075-e465-439f-ac27-63a19d7f8ec9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms\" (UID: \"f5128075-e465-439f-ac27-63a19d7f8ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"
Sep 11 12:44:05.423298 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:05.423178    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf5kd\" (UniqueName: \"kubernetes.io/projected/f5128075-e465-439f-ac27-63a19d7f8ec9-kube-api-access-bf5kd\") pod \"maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms\" (UID: \"f5128075-e465-439f-ac27-63a19d7f8ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"
Sep 11 12:44:05.423298 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:05.423258    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5128075-e465-439f-ac27-63a19d7f8ec9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms\" (UID: \"f5128075-e465-439f-ac27-63a19d7f8ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"
Sep 11 12:44:05.423424 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:44:05.423365    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-serving-cert: secret "maas-api-e2e-ipp-b-d5ae2a-serving-cert" not found
Sep 11 12:44:05.423465 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:44:05.423433    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5128075-e465-439f-ac27-63a19d7f8ec9-maas-api-tls podName:f5128075-e465-439f-ac27-63a19d7f8ec9 nodeName:}" failed. No retries permitted until 2026-09-11 12:44:05.92341053 +0000 UTC m=+1777.486190271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f5128075-e465-439f-ac27-63a19d7f8ec9-maas-api-tls") pod "maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms" (UID: "f5128075-e465-439f-ac27-63a19d7f8ec9") : secret "maas-api-e2e-ipp-b-d5ae2a-serving-cert" not found
Sep 11 12:44:05.423885 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:05.423866    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5128075-e465-439f-ac27-63a19d7f8ec9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms\" (UID: \"f5128075-e465-439f-ac27-63a19d7f8ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"
Sep 11 12:44:05.425486 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:05.425468    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5128075-e465-439f-ac27-63a19d7f8ec9-metrics-certs\") pod \"maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms\" (UID: \"f5128075-e465-439f-ac27-63a19d7f8ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"
Sep 11 12:44:05.431632 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:05.431615    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf5kd\" (UniqueName: \"kubernetes.io/projected/f5128075-e465-439f-ac27-63a19d7f8ec9-kube-api-access-bf5kd\") pod \"maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms\" (UID: \"f5128075-e465-439f-ac27-63a19d7f8ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"
Sep 11 12:44:05.927900 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:05.927868    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5128075-e465-439f-ac27-63a19d7f8ec9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms\" (UID: \"f5128075-e465-439f-ac27-63a19d7f8ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"
Sep 11 12:44:05.930242 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:05.930219    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5128075-e465-439f-ac27-63a19d7f8ec9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms\" (UID: \"f5128075-e465-439f-ac27-63a19d7f8ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"
Sep 11 12:44:06.000181 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:06.000148    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn"]
Sep 11 12:44:06.004195 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:06.004170    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn"
Sep 11 12:44:06.007060 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:06.007042    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d5ae2a-dockercfg-4tbqk\""
Sep 11 12:44:06.007150 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:06.007045    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d5ae2a\""
Sep 11 12:44:06.015736 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:06.015715    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn"]
Sep 11 12:44:06.116731 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:06.116706    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"
Sep 11 12:44:06.129589 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:06.129563    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56dbcd44-a271-4389-a6b2-e018fa5e0867-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn\" (UID: \"56dbcd44-a271-4389-a6b2-e018fa5e0867\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn"
Sep 11 12:44:06.129700 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:06.129609    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzvkz\" (UniqueName: \"kubernetes.io/projected/56dbcd44-a271-4389-a6b2-e018fa5e0867-kube-api-access-rzvkz\") pod \"payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn\" (UID: \"56dbcd44-a271-4389-a6b2-e018fa5e0867\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn"
Sep 11 12:44:06.230958 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:06.230925    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56dbcd44-a271-4389-a6b2-e018fa5e0867-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn\" (UID: \"56dbcd44-a271-4389-a6b2-e018fa5e0867\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn"
Sep 11 12:44:06.231343 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:06.230979    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzvkz\" (UniqueName: \"kubernetes.io/projected/56dbcd44-a271-4389-a6b2-e018fa5e0867-kube-api-access-rzvkz\") pod \"payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn\" (UID: \"56dbcd44-a271-4389-a6b2-e018fa5e0867\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn"
Sep 11 12:44:06.231682 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:06.231660    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56dbcd44-a271-4389-a6b2-e018fa5e0867-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn\" (UID: \"56dbcd44-a271-4389-a6b2-e018fa5e0867\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn"
Sep 11 12:44:06.238417 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:06.238395    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzvkz\" (UniqueName: \"kubernetes.io/projected/56dbcd44-a271-4389-a6b2-e018fa5e0867-kube-api-access-rzvkz\") pod \"payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn\" (UID: \"56dbcd44-a271-4389-a6b2-e018fa5e0867\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn"
Sep 11 12:44:06.315138 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:06.315103    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn"
Sep 11 12:44:06.391411 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:06.391379    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"]
Sep 11 12:44:06.392743 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:44:06.392720    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5128075_e465_439f_ac27_63a19d7f8ec9.slice/crio-b34b6c08a1aa6509fc8b19d3f8662749eb80a082a1d5882e2a25b115139cc3aa WatchSource:0}: Error finding container b34b6c08a1aa6509fc8b19d3f8662749eb80a082a1d5882e2a25b115139cc3aa: Status 404 returned error can't find the container with id b34b6c08a1aa6509fc8b19d3f8662749eb80a082a1d5882e2a25b115139cc3aa
Sep 11 12:44:06.585250 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:06.585216    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn"]
Sep 11 12:44:06.587602 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:44:06.587563    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56dbcd44_a271_4389_a6b2_e018fa5e0867.slice/crio-d229c6908a909d2e32c1e284477994d58bcde4a9365f6336f30256097c58cf4f WatchSource:0}: Error finding container d229c6908a909d2e32c1e284477994d58bcde4a9365f6336f30256097c58cf4f: Status 404 returned error can't find the container with id d229c6908a909d2e32c1e284477994d58bcde4a9365f6336f30256097c58cf4f
Sep 11 12:44:07.330287 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:07.330252    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms" event={"ID":"f5128075-e465-439f-ac27-63a19d7f8ec9","Type":"ContainerStarted","Data":"22c82194c17f26a1703c3b71ad00652dad29f6018e12f92599205d66e78eb359"}
Sep 11 12:44:07.330797 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:07.330310    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms" event={"ID":"f5128075-e465-439f-ac27-63a19d7f8ec9","Type":"ContainerStarted","Data":"b34b6c08a1aa6509fc8b19d3f8662749eb80a082a1d5882e2a25b115139cc3aa"}
Sep 11 12:44:07.330797 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:07.330415    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"
Sep 11 12:44:07.331877 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:07.331847    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn" event={"ID":"56dbcd44-a271-4389-a6b2-e018fa5e0867","Type":"ContainerStarted","Data":"c025dd8b5a9a24387b450b0bcb2c68b1509bc1d7fececd41919e763b4abfadbe"}
Sep 11 12:44:07.331975 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:07.331898    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn" event={"ID":"56dbcd44-a271-4389-a6b2-e018fa5e0867","Type":"ContainerStarted","Data":"d229c6908a909d2e32c1e284477994d58bcde4a9365f6336f30256097c58cf4f"}
Sep 11 12:44:07.331975 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:07.331921    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn"
Sep 11 12:44:07.347513 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:07.347468    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms" podStartSLOduration=2.024847851 podStartE2EDuration="2.347455266s" totalImagesPullingTime="322.607415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:44:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:44:07.346071372 +0000 UTC m=+1778.908851137" watchObservedRunningTime="2026-09-11 12:44:07.347455266 +0000 UTC m=+1778.910235029"
Sep 11 12:44:07.362685 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:07.362639    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn" podStartSLOduration=2.362625911 podStartE2EDuration="2.362625911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:44:07.360965167 +0000 UTC m=+1778.923744930" watchObservedRunningTime="2026-09-11 12:44:07.362625911 +0000 UTC m=+1778.925405674"
Sep 11 12:44:10.843412 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:10.843377    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:44:13.342624 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:13.342595    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"
Sep 11 12:44:18.339204 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:18.339163    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn"
Sep 11 12:44:28.985564 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:28.985538    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcfm6_88402687-4e5b-411c-be7a-4cb955f90425/ovn-acl-logging/0.log"
Sep 11 12:44:28.991549 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:28.991531    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcfm6_88402687-4e5b-411c-be7a-4cb955f90425/ovn-acl-logging/0.log"
Sep 11 12:44:38.407289 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.407252    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"]
Sep 11 12:44:38.410579 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.410562    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:44:38.413528 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.413504    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0a1087\"/\"default-dockercfg-nxfx5\""
Sep 11 12:44:38.414480 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.414460    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0a1087\"/\"openshift-service-ca.crt\""
Sep 11 12:44:38.414480 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.414470    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0a1087\"/\"kube-root-ca.crt\""
Sep 11 12:44:38.414632 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.414527    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0a1087\"/\"ipp-route-0a1087-kserve-self-signed-certs\""
Sep 11 12:44:38.424275 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.424254    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"]
Sep 11 12:44:38.501597 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.501564    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-dshm\") pod \"ipp-route-0a1087-kserve-7b648854ff-sl66q\" (UID: \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\") " pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:44:38.501754 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.501622    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-model-cache\") pod \"ipp-route-0a1087-kserve-7b648854ff-sl66q\" (UID: \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\") " pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:44:38.501754 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.501669    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-home\") pod \"ipp-route-0a1087-kserve-7b648854ff-sl66q\" (UID: \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\") " pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:44:38.501754 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.501700    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk7d6\" (UniqueName: \"kubernetes.io/projected/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-kube-api-access-vk7d6\") pod \"ipp-route-0a1087-kserve-7b648854ff-sl66q\" (UID: \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\") " pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:44:38.501865 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.501752    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-tls-certs\") pod \"ipp-route-0a1087-kserve-7b648854ff-sl66q\" (UID: \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\") " pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:44:38.501865 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.501804    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-tmp-dir\") pod \"ipp-route-0a1087-kserve-7b648854ff-sl66q\" (UID: \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\") " pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:44:38.602550 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.602512    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk7d6\" (UniqueName: \"kubernetes.io/projected/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-kube-api-access-vk7d6\") pod \"ipp-route-0a1087-kserve-7b648854ff-sl66q\" (UID: \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\") " pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:44:38.602550 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.602555    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-tls-certs\") pod \"ipp-route-0a1087-kserve-7b648854ff-sl66q\" (UID: \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\") " pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:44:38.602782 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.602598    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-tmp-dir\") pod \"ipp-route-0a1087-kserve-7b648854ff-sl66q\" (UID: \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\") " pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:44:38.602782 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.602632    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-dshm\") pod \"ipp-route-0a1087-kserve-7b648854ff-sl66q\" (UID: \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\") " pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:44:38.602899 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.602777    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-model-cache\") pod \"ipp-route-0a1087-kserve-7b648854ff-sl66q\" (UID: \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\") " pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:44:38.602899 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.602843    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-home\") pod \"ipp-route-0a1087-kserve-7b648854ff-sl66q\" (UID: \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\") " pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:44:38.603088 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.603064    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-tmp-dir\") pod \"ipp-route-0a1087-kserve-7b648854ff-sl66q\" (UID: \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\") " pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:44:38.603178 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.603130    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-model-cache\") pod \"ipp-route-0a1087-kserve-7b648854ff-sl66q\" (UID: \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\") " pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:44:38.603178 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.603170    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-home\") pod \"ipp-route-0a1087-kserve-7b648854ff-sl66q\" (UID: \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\") " pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:44:38.604887 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.604864    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-dshm\") pod \"ipp-route-0a1087-kserve-7b648854ff-sl66q\" (UID: \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\") " pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:44:38.605342 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.605319    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-tls-certs\") pod \"ipp-route-0a1087-kserve-7b648854ff-sl66q\" (UID: \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\") " pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:44:38.612464 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.612437    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk7d6\" (UniqueName: \"kubernetes.io/projected/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-kube-api-access-vk7d6\") pod \"ipp-route-0a1087-kserve-7b648854ff-sl66q\" (UID: \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\") " pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:44:38.719520 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.719449    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:44:38.996003 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:38.995972    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"]
Sep 11 12:44:38.998835 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:44:38.998807    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c130d3d_73c4_495e_a903_2ebda2a0f3d9.slice/crio-3859f150d53a3f62da87d1031e9fc459c67c9d0827984a913863235adce1cb41 WatchSource:0}: Error finding container 3859f150d53a3f62da87d1031e9fc459c67c9d0827984a913863235adce1cb41: Status 404 returned error can't find the container with id 3859f150d53a3f62da87d1031e9fc459c67c9d0827984a913863235adce1cb41
Sep 11 12:44:39.488894 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:39.488858    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q" event={"ID":"8c130d3d-73c4-495e-a903-2ebda2a0f3d9","Type":"ContainerStarted","Data":"dd702e1e3ac33d81e7adf0eef044c73c8207f2bcfbf6d602c1e56c44edbd9f0f"}
Sep 11 12:44:39.489320 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:39.488899    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q" event={"ID":"8c130d3d-73c4-495e-a903-2ebda2a0f3d9","Type":"ContainerStarted","Data":"3859f150d53a3f62da87d1031e9fc459c67c9d0827984a913863235adce1cb41"}
Sep 11 12:44:39.508942 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:39.508897    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q" podStartSLOduration=1.5088822290000001 podStartE2EDuration="1.508882229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:44:39.507028792 +0000 UTC m=+1811.069808556" watchObservedRunningTime="2026-09-11 12:44:39.508882229 +0000 UTC m=+1811.071661992"
Sep 11 12:44:42.048352 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:42.048319    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:44:48.720243 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:48.720173    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:44:48.720645 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:48.720266    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:44:48.732738 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:48.732710    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:44:49.549124 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:44:49.549097    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:45:00.141106 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:00.141072    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa824-29818845-wfq7x"]
Sep 11 12:45:00.145589 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:00.145569    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa824-29818845-wfq7x"
Sep 11 12:45:00.147920 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:00.147900    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\""
Sep 11 12:45:00.153520 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:00.153450    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa824-29818845-wfq7x"]
Sep 11 12:45:00.157437 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:00.157416    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa824-29818845-wfq7x"
Sep 11 12:45:00.428475 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:00.428450    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa824-29818845-wfq7x"]
Sep 11 12:45:00.430920 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:45:00.430892    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f1aff2_550b_4dd2_bbe3_7acdd08acf76.slice/crio-ce644a4d0abca11cf5699e40bfe60a66873515bf6815f7252d2ddc0cca5708f5 WatchSource:0}: Error finding container ce644a4d0abca11cf5699e40bfe60a66873515bf6815f7252d2ddc0cca5708f5: Status 404 returned error can't find the container with id ce644a4d0abca11cf5699e40bfe60a66873515bf6815f7252d2ddc0cca5708f5
Sep 11 12:45:00.601504 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:00.601474    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa824-29818845-wfq7x" event={"ID":"b6f1aff2-550b-4dd2-bbe3-7acdd08acf76","Type":"ContainerStarted","Data":"ce644a4d0abca11cf5699e40bfe60a66873515bf6815f7252d2ddc0cca5708f5"}
Sep 11 12:45:01.607976 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:01.607894    2561 generic.go:356] "Generic (PLEG): container finished" podID="b6f1aff2-550b-4dd2-bbe3-7acdd08acf76" containerID="391c7aaaaaf103be7801e5a950af39b92f3783485272ab161a3ac53e74c2e77b" exitCode=0
Sep 11 12:45:01.607976 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:01.607929    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa824-29818845-wfq7x" event={"ID":"b6f1aff2-550b-4dd2-bbe3-7acdd08acf76","Type":"ContainerDied","Data":"391c7aaaaaf103be7801e5a950af39b92f3783485272ab161a3ac53e74c2e77b"}
Sep 11 12:45:02.740024 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:02.739999    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa824-29818845-wfq7x"
Sep 11 12:45:03.617756 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:03.617728    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa824-29818845-wfq7x"
Sep 11 12:45:03.617931 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:03.617774    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa824-29818845-wfq7x" event={"ID":"b6f1aff2-550b-4dd2-bbe3-7acdd08acf76","Type":"ContainerDied","Data":"ce644a4d0abca11cf5699e40bfe60a66873515bf6815f7252d2ddc0cca5708f5"}
Sep 11 12:45:03.617931 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:03.617822    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce644a4d0abca11cf5699e40bfe60a66873515bf6815f7252d2ddc0cca5708f5"
Sep 11 12:45:19.240522 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:19.240483    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:45:25.712922 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:25.712883    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv_62a6ff6a-3c11-4036-9d57-636a55aa0ca1/payload-processing/0.log"
Sep 11 12:45:30.730527 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:30.730493    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"]
Sep 11 12:45:30.732902 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:30.731028    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6f1aff2-550b-4dd2-bbe3-7acdd08acf76" containerName="cleanup"
Sep 11 12:45:30.732902 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:30.731044    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6f1aff2-550b-4dd2-bbe3-7acdd08acf76" containerName="cleanup"
Sep 11 12:45:30.732902 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:30.731119    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6f1aff2-550b-4dd2-bbe3-7acdd08acf76" containerName="cleanup"
Sep 11 12:45:30.733937 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:30.733917    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"
Sep 11 12:45:30.736991 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:30.736973    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-f92540-serving-cert\""
Sep 11 12:45:30.755595 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:30.755576    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/999e3562-6f74-4e8b-b40e-c863102a3d92-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr\" (UID: \"999e3562-6f74-4e8b-b40e-c863102a3d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"
Sep 11 12:45:30.755707 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:30.755625    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/999e3562-6f74-4e8b-b40e-c863102a3d92-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr\" (UID: \"999e3562-6f74-4e8b-b40e-c863102a3d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"
Sep 11 12:45:30.755774 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:30.755734    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tjgz\" (UniqueName: \"kubernetes.io/projected/999e3562-6f74-4e8b-b40e-c863102a3d92-kube-api-access-2tjgz\") pod \"maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr\" (UID: \"999e3562-6f74-4e8b-b40e-c863102a3d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"
Sep 11 12:45:30.755834 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:30.755811    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/999e3562-6f74-4e8b-b40e-c863102a3d92-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr\" (UID: \"999e3562-6f74-4e8b-b40e-c863102a3d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"
Sep 11 12:45:30.761642 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:30.761616    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"]
Sep 11 12:45:30.856522 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:30.856496    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/999e3562-6f74-4e8b-b40e-c863102a3d92-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr\" (UID: \"999e3562-6f74-4e8b-b40e-c863102a3d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"
Sep 11 12:45:30.856690 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:30.856535    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/999e3562-6f74-4e8b-b40e-c863102a3d92-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr\" (UID: \"999e3562-6f74-4e8b-b40e-c863102a3d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"
Sep 11 12:45:30.856690 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:30.856564    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/999e3562-6f74-4e8b-b40e-c863102a3d92-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr\" (UID: \"999e3562-6f74-4e8b-b40e-c863102a3d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"
Sep 11 12:45:30.856690 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:30.856606    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tjgz\" (UniqueName: \"kubernetes.io/projected/999e3562-6f74-4e8b-b40e-c863102a3d92-kube-api-access-2tjgz\") pod \"maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr\" (UID: \"999e3562-6f74-4e8b-b40e-c863102a3d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"
Sep 11 12:45:30.856690 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:45:30.856666    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-serving-cert: secret "maas-api-e2e-ipp-cleanup-f92540-serving-cert" not found
Sep 11 12:45:30.856918 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:45:30.856751    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/999e3562-6f74-4e8b-b40e-c863102a3d92-maas-api-tls podName:999e3562-6f74-4e8b-b40e-c863102a3d92 nodeName:}" failed. No retries permitted until 2026-09-11 12:45:31.356730407 +0000 UTC m=+1862.919510150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/999e3562-6f74-4e8b-b40e-c863102a3d92-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr" (UID: "999e3562-6f74-4e8b-b40e-c863102a3d92") : secret "maas-api-e2e-ipp-cleanup-f92540-serving-cert" not found
Sep 11 12:45:30.857145 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:30.857123    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/999e3562-6f74-4e8b-b40e-c863102a3d92-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr\" (UID: \"999e3562-6f74-4e8b-b40e-c863102a3d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"
Sep 11 12:45:30.858948 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:30.858918    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/999e3562-6f74-4e8b-b40e-c863102a3d92-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr\" (UID: \"999e3562-6f74-4e8b-b40e-c863102a3d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"
Sep 11 12:45:30.878244 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:30.878220    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tjgz\" (UniqueName: \"kubernetes.io/projected/999e3562-6f74-4e8b-b40e-c863102a3d92-kube-api-access-2tjgz\") pod \"maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr\" (UID: \"999e3562-6f74-4e8b-b40e-c863102a3d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"
Sep 11 12:45:31.361470 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:31.361437    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/999e3562-6f74-4e8b-b40e-c863102a3d92-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr\" (UID: \"999e3562-6f74-4e8b-b40e-c863102a3d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"
Sep 11 12:45:31.363852 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:31.363819    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/999e3562-6f74-4e8b-b40e-c863102a3d92-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr\" (UID: \"999e3562-6f74-4e8b-b40e-c863102a3d92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"
Sep 11 12:45:31.644451 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:31.644371    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"
Sep 11 12:45:31.704061 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:31.704031    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6"]
Sep 11 12:45:31.708753 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:31.708731    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6"
Sep 11 12:45:31.712353 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:31.712303    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-f92540-dockercfg-rfwwg\""
Sep 11 12:45:31.712465 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:31.712369    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-f92540\""
Sep 11 12:45:31.726315 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:31.726291    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6"]
Sep 11 12:45:31.765316 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:31.765276    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww5rf\" (UniqueName: \"kubernetes.io/projected/18320f23-b4a4-491e-bb68-27c3983eca0f-kube-api-access-ww5rf\") pod \"payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6\" (UID: \"18320f23-b4a4-491e-bb68-27c3983eca0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6"
Sep 11 12:45:31.765682 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:31.765341    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18320f23-b4a4-491e-bb68-27c3983eca0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6\" (UID: \"18320f23-b4a4-491e-bb68-27c3983eca0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6"
Sep 11 12:45:31.866035 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:31.866003    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww5rf\" (UniqueName: \"kubernetes.io/projected/18320f23-b4a4-491e-bb68-27c3983eca0f-kube-api-access-ww5rf\") pod \"payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6\" (UID: \"18320f23-b4a4-491e-bb68-27c3983eca0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6"
Sep 11 12:45:31.866209 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:31.866067    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18320f23-b4a4-491e-bb68-27c3983eca0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6\" (UID: \"18320f23-b4a4-491e-bb68-27c3983eca0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6"
Sep 11 12:45:31.866735 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:31.866716    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18320f23-b4a4-491e-bb68-27c3983eca0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6\" (UID: \"18320f23-b4a4-491e-bb68-27c3983eca0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6"
Sep 11 12:45:31.880599 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:31.880573    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww5rf\" (UniqueName: \"kubernetes.io/projected/18320f23-b4a4-491e-bb68-27c3983eca0f-kube-api-access-ww5rf\") pod \"payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6\" (UID: \"18320f23-b4a4-491e-bb68-27c3983eca0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6"
Sep 11 12:45:31.921824 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:31.921801    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"]
Sep 11 12:45:31.923788 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:45:31.923760    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod999e3562_6f74_4e8b_b40e_c863102a3d92.slice/crio-d1051ccd75a08f949693f88d8994c279a35ba34cf698a3667e7ba0767902cc12 WatchSource:0}: Error finding container d1051ccd75a08f949693f88d8994c279a35ba34cf698a3667e7ba0767902cc12: Status 404 returned error can't find the container with id d1051ccd75a08f949693f88d8994c279a35ba34cf698a3667e7ba0767902cc12
Sep 11 12:45:32.022075 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:32.022044    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6"
Sep 11 12:45:32.302239 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:32.302215    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6"]
Sep 11 12:45:32.304147 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:45:32.304124    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18320f23_b4a4_491e_bb68_27c3983eca0f.slice/crio-2ffc9b4ec007364973bef6e38866afce1d0cb80fdb5b82b406809d2270c7f92e WatchSource:0}: Error finding container 2ffc9b4ec007364973bef6e38866afce1d0cb80fdb5b82b406809d2270c7f92e: Status 404 returned error can't find the container with id 2ffc9b4ec007364973bef6e38866afce1d0cb80fdb5b82b406809d2270c7f92e
Sep 11 12:45:32.765262 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:32.765225    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr" event={"ID":"999e3562-6f74-4e8b-b40e-c863102a3d92","Type":"ContainerStarted","Data":"157b5369904e7ddc4bd8f6e4fdfcabb985510a55ba17dec761066a9b713bd84a"}
Sep 11 12:45:32.765445 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:32.765280    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr" event={"ID":"999e3562-6f74-4e8b-b40e-c863102a3d92","Type":"ContainerStarted","Data":"d1051ccd75a08f949693f88d8994c279a35ba34cf698a3667e7ba0767902cc12"}
Sep 11 12:45:32.765445 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:32.765313    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"
Sep 11 12:45:32.766869 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:32.766845    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6" event={"ID":"18320f23-b4a4-491e-bb68-27c3983eca0f","Type":"ContainerStarted","Data":"abcf3c21bb5f85f244d62767d98c662cb74892c2c27f0a1cdd4a49e0ef9aa136"}
Sep 11 12:45:32.766965 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:32.766892    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6" event={"ID":"18320f23-b4a4-491e-bb68-27c3983eca0f","Type":"ContainerStarted","Data":"2ffc9b4ec007364973bef6e38866afce1d0cb80fdb5b82b406809d2270c7f92e"}
Sep 11 12:45:32.766965 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:32.766923    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6"
Sep 11 12:45:32.789225 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:32.789165    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr" podStartSLOduration=2.20390126 podStartE2EDuration="2.789151861s" totalImagesPullingTime="585.250601ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:45:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:45:32.786581053 +0000 UTC m=+1864.349360816" watchObservedRunningTime="2026-09-11 12:45:32.789151861 +0000 UTC m=+1864.351931625"
Sep 11 12:45:32.807511 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:32.807425    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6" podStartSLOduration=1.807412533 podStartE2EDuration="1.807412533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:45:32.807060512 +0000 UTC m=+1864.369840275" watchObservedRunningTime="2026-09-11 12:45:32.807412533 +0000 UTC m=+1864.370192295"
Sep 11 12:45:37.142344 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:37.142306    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:45:38.778060 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:38.778029    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"
Sep 11 12:45:43.774827 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:45:43.774799    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6"
Sep 11 12:46:01.380820 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.380745    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"]
Sep 11 12:46:01.381300 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.380937    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr" podUID="999e3562-6f74-4e8b-b40e-c863102a3d92" containerName="maas-api" containerID="cri-o://157b5369904e7ddc4bd8f6e4fdfcabb985510a55ba17dec761066a9b713bd84a" gracePeriod=30
Sep 11 12:46:01.449702 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.449672    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6"]
Sep 11 12:46:01.449934 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.449887    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6" podUID="18320f23-b4a4-491e-bb68-27c3983eca0f" containerName="payload-pre-processing" containerID="cri-o://abcf3c21bb5f85f244d62767d98c662cb74892c2c27f0a1cdd4a49e0ef9aa136" gracePeriod=30
Sep 11 12:46:01.689251 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.689223    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"
Sep 11 12:46:01.718356 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.718334    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6"
Sep 11 12:46:01.835759 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.835723    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tjgz\" (UniqueName: \"kubernetes.io/projected/999e3562-6f74-4e8b-b40e-c863102a3d92-kube-api-access-2tjgz\") pod \"999e3562-6f74-4e8b-b40e-c863102a3d92\" (UID: \"999e3562-6f74-4e8b-b40e-c863102a3d92\") "
Sep 11 12:46:01.835940 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.835800    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/999e3562-6f74-4e8b-b40e-c863102a3d92-openshift-service-ca\") pod \"999e3562-6f74-4e8b-b40e-c863102a3d92\" (UID: \"999e3562-6f74-4e8b-b40e-c863102a3d92\") "
Sep 11 12:46:01.835940 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.835835    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/999e3562-6f74-4e8b-b40e-c863102a3d92-maas-api-tls\") pod \"999e3562-6f74-4e8b-b40e-c863102a3d92\" (UID: \"999e3562-6f74-4e8b-b40e-c863102a3d92\") "
Sep 11 12:46:01.835940 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.835851    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18320f23-b4a4-491e-bb68-27c3983eca0f-plugins-config-volume\") pod \"18320f23-b4a4-491e-bb68-27c3983eca0f\" (UID: \"18320f23-b4a4-491e-bb68-27c3983eca0f\") "
Sep 11 12:46:01.835940 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.835871    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/999e3562-6f74-4e8b-b40e-c863102a3d92-metrics-certs\") pod \"999e3562-6f74-4e8b-b40e-c863102a3d92\" (UID: \"999e3562-6f74-4e8b-b40e-c863102a3d92\") "
Sep 11 12:46:01.835940 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.835899    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww5rf\" (UniqueName: \"kubernetes.io/projected/18320f23-b4a4-491e-bb68-27c3983eca0f-kube-api-access-ww5rf\") pod \"18320f23-b4a4-491e-bb68-27c3983eca0f\" (UID: \"18320f23-b4a4-491e-bb68-27c3983eca0f\") "
Sep 11 12:46:01.836286 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.836257    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999e3562-6f74-4e8b-b40e-c863102a3d92-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "999e3562-6f74-4e8b-b40e-c863102a3d92" (UID: "999e3562-6f74-4e8b-b40e-c863102a3d92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:46:01.836356 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.836311    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18320f23-b4a4-491e-bb68-27c3983eca0f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "18320f23-b4a4-491e-bb68-27c3983eca0f" (UID: "18320f23-b4a4-491e-bb68-27c3983eca0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:46:01.837934 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.837914    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/999e3562-6f74-4e8b-b40e-c863102a3d92-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "999e3562-6f74-4e8b-b40e-c863102a3d92" (UID: "999e3562-6f74-4e8b-b40e-c863102a3d92"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:46:01.838102 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.838081    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18320f23-b4a4-491e-bb68-27c3983eca0f-kube-api-access-ww5rf" (OuterVolumeSpecName: "kube-api-access-ww5rf") pod "18320f23-b4a4-491e-bb68-27c3983eca0f" (UID: "18320f23-b4a4-491e-bb68-27c3983eca0f"). InnerVolumeSpecName "kube-api-access-ww5rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:46:01.838395 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.838372    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/999e3562-6f74-4e8b-b40e-c863102a3d92-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "999e3562-6f74-4e8b-b40e-c863102a3d92" (UID: "999e3562-6f74-4e8b-b40e-c863102a3d92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:46:01.838395 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.838380    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999e3562-6f74-4e8b-b40e-c863102a3d92-kube-api-access-2tjgz" (OuterVolumeSpecName: "kube-api-access-2tjgz") pod "999e3562-6f74-4e8b-b40e-c863102a3d92" (UID: "999e3562-6f74-4e8b-b40e-c863102a3d92"). InnerVolumeSpecName "kube-api-access-2tjgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:46:01.913799 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.913722    2561 generic.go:356] "Generic (PLEG): container finished" podID="18320f23-b4a4-491e-bb68-27c3983eca0f" containerID="abcf3c21bb5f85f244d62767d98c662cb74892c2c27f0a1cdd4a49e0ef9aa136" exitCode=0
Sep 11 12:46:01.913799 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.913761    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6"
Sep 11 12:46:01.914009 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.913803    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6" event={"ID":"18320f23-b4a4-491e-bb68-27c3983eca0f","Type":"ContainerDied","Data":"abcf3c21bb5f85f244d62767d98c662cb74892c2c27f0a1cdd4a49e0ef9aa136"}
Sep 11 12:46:01.914009 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.913858    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6" event={"ID":"18320f23-b4a4-491e-bb68-27c3983eca0f","Type":"ContainerDied","Data":"2ffc9b4ec007364973bef6e38866afce1d0cb80fdb5b82b406809d2270c7f92e"}
Sep 11 12:46:01.914009 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.913879    2561 scope.go:117] "RemoveContainer" containerID="abcf3c21bb5f85f244d62767d98c662cb74892c2c27f0a1cdd4a49e0ef9aa136"
Sep 11 12:46:01.915366 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.915297    2561 generic.go:356] "Generic (PLEG): container finished" podID="999e3562-6f74-4e8b-b40e-c863102a3d92" containerID="157b5369904e7ddc4bd8f6e4fdfcabb985510a55ba17dec761066a9b713bd84a" exitCode=0
Sep 11 12:46:01.915366 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.915346    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr" event={"ID":"999e3562-6f74-4e8b-b40e-c863102a3d92","Type":"ContainerDied","Data":"157b5369904e7ddc4bd8f6e4fdfcabb985510a55ba17dec761066a9b713bd84a"}
Sep 11 12:46:01.915559 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.915374    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"
Sep 11 12:46:01.915559 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.915387    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr" event={"ID":"999e3562-6f74-4e8b-b40e-c863102a3d92","Type":"ContainerDied","Data":"d1051ccd75a08f949693f88d8994c279a35ba34cf698a3667e7ba0767902cc12"}
Sep 11 12:46:01.923695 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.923677    2561 scope.go:117] "RemoveContainer" containerID="abcf3c21bb5f85f244d62767d98c662cb74892c2c27f0a1cdd4a49e0ef9aa136"
Sep 11 12:46:01.923985 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:46:01.923963    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abcf3c21bb5f85f244d62767d98c662cb74892c2c27f0a1cdd4a49e0ef9aa136\": container with ID starting with abcf3c21bb5f85f244d62767d98c662cb74892c2c27f0a1cdd4a49e0ef9aa136 not found: ID does not exist" containerID="abcf3c21bb5f85f244d62767d98c662cb74892c2c27f0a1cdd4a49e0ef9aa136"
Sep 11 12:46:01.924057 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.923995    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abcf3c21bb5f85f244d62767d98c662cb74892c2c27f0a1cdd4a49e0ef9aa136"} err="failed to get container status \"abcf3c21bb5f85f244d62767d98c662cb74892c2c27f0a1cdd4a49e0ef9aa136\": rpc error: code = NotFound desc = could not find container \"abcf3c21bb5f85f244d62767d98c662cb74892c2c27f0a1cdd4a49e0ef9aa136\": container with ID starting with abcf3c21bb5f85f244d62767d98c662cb74892c2c27f0a1cdd4a49e0ef9aa136 not found: ID does not exist"
Sep 11 12:46:01.924057 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.924018    2561 scope.go:117] "RemoveContainer" containerID="157b5369904e7ddc4bd8f6e4fdfcabb985510a55ba17dec761066a9b713bd84a"
Sep 11 12:46:01.931988 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.931973    2561 scope.go:117] "RemoveContainer" containerID="157b5369904e7ddc4bd8f6e4fdfcabb985510a55ba17dec761066a9b713bd84a"
Sep 11 12:46:01.932250 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:46:01.932232    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"157b5369904e7ddc4bd8f6e4fdfcabb985510a55ba17dec761066a9b713bd84a\": container with ID starting with 157b5369904e7ddc4bd8f6e4fdfcabb985510a55ba17dec761066a9b713bd84a not found: ID does not exist" containerID="157b5369904e7ddc4bd8f6e4fdfcabb985510a55ba17dec761066a9b713bd84a"
Sep 11 12:46:01.932307 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.932255    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"157b5369904e7ddc4bd8f6e4fdfcabb985510a55ba17dec761066a9b713bd84a"} err="failed to get container status \"157b5369904e7ddc4bd8f6e4fdfcabb985510a55ba17dec761066a9b713bd84a\": rpc error: code = NotFound desc = could not find container \"157b5369904e7ddc4bd8f6e4fdfcabb985510a55ba17dec761066a9b713bd84a\": container with ID starting with 157b5369904e7ddc4bd8f6e4fdfcabb985510a55ba17dec761066a9b713bd84a not found: ID does not exist"
Sep 11 12:46:01.936987 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.936966    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tjgz\" (UniqueName: \"kubernetes.io/projected/999e3562-6f74-4e8b-b40e-c863102a3d92-kube-api-access-2tjgz\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:46:01.937065 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.936988    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/999e3562-6f74-4e8b-b40e-c863102a3d92-openshift-service-ca\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:46:01.937065 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.936998    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/999e3562-6f74-4e8b-b40e-c863102a3d92-maas-api-tls\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:46:01.937065 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.937008    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18320f23-b4a4-491e-bb68-27c3983eca0f-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:46:01.937065 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.937020    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/999e3562-6f74-4e8b-b40e-c863102a3d92-metrics-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:46:01.937065 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.937032    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww5rf\" (UniqueName: \"kubernetes.io/projected/18320f23-b4a4-491e-bb68-27c3983eca0f-kube-api-access-ww5rf\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:46:01.937584 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.937566    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6"]
Sep 11 12:46:01.946910 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.946885    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f92540-5f67cf7694-jdfw6"]
Sep 11 12:46:01.958767 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.958746    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"]
Sep 11 12:46:01.965132 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:01.965111    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f92540-54688c6d76-hl5pr"]
Sep 11 12:46:02.931485 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:02.931454    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18320f23-b4a4-491e-bb68-27c3983eca0f" path="/var/lib/kubelet/pods/18320f23-b4a4-491e-bb68-27c3983eca0f/volumes"
Sep 11 12:46:02.931842 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:02.931822    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="999e3562-6f74-4e8b-b40e-c863102a3d92" path="/var/lib/kubelet/pods/999e3562-6f74-4e8b-b40e-c863102a3d92/volumes"
Sep 11 12:46:05.424628 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:05.424595    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:46:24.833093 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:24.833054    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:46:27.465988 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:27.465951    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh"]
Sep 11 12:46:27.466477 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:27.466201    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh" podUID="28f97f04-3cc3-4fa0-99ce-471ed9a389a2" containerName="maas-api" containerID="cri-o://e253aaf8524bb84969349e766af52e61f86a2f73bbc6f3107ba336d825b96d3e" gracePeriod=30
Sep 11 12:46:27.527734 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:27.527700    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv"]
Sep 11 12:46:27.528012 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:27.527984    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv" podUID="62a6ff6a-3c11-4036-9d57-636a55aa0ca1" containerName="payload-processing" containerID="cri-o://1b1cf1d126d557ce970b8dcff3468e5563273c8abd825480bb77cb0ff98a8cfa" gracePeriod=30
Sep 11 12:46:27.788300 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:27.788280    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh"
Sep 11 12:46:27.791198 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:27.791135    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv"
Sep 11 12:46:27.849316 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:27.849284    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a6ff6a-3c11-4036-9d57-636a55aa0ca1-plugins-config-volume\") pod \"62a6ff6a-3c11-4036-9d57-636a55aa0ca1\" (UID: \"62a6ff6a-3c11-4036-9d57-636a55aa0ca1\") "
Sep 11 12:46:27.849489 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:27.849342    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f97f04-3cc3-4fa0-99ce-471ed9a389a2-openshift-service-ca\") pod \"28f97f04-3cc3-4fa0-99ce-471ed9a389a2\" (UID: \"28f97f04-3cc3-4fa0-99ce-471ed9a389a2\") "
Sep 11 12:46:27.849489 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:27.849369    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f97f04-3cc3-4fa0-99ce-471ed9a389a2-metrics-certs\") pod \"28f97f04-3cc3-4fa0-99ce-471ed9a389a2\" (UID: \"28f97f04-3cc3-4fa0-99ce-471ed9a389a2\") "
Sep 11 12:46:27.849489 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:27.849418    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbjbp\" (UniqueName: \"kubernetes.io/projected/28f97f04-3cc3-4fa0-99ce-471ed9a389a2-kube-api-access-mbjbp\") pod \"28f97f04-3cc3-4fa0-99ce-471ed9a389a2\" (UID: \"28f97f04-3cc3-4fa0-99ce-471ed9a389a2\") "
Sep 11 12:46:27.849489 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:27.849439    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlwkg\" (UniqueName: \"kubernetes.io/projected/62a6ff6a-3c11-4036-9d57-636a55aa0ca1-kube-api-access-nlwkg\") pod \"62a6ff6a-3c11-4036-9d57-636a55aa0ca1\" (UID: \"62a6ff6a-3c11-4036-9d57-636a55aa0ca1\") "
Sep 11 12:46:27.849489 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:27.849461    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f97f04-3cc3-4fa0-99ce-471ed9a389a2-maas-api-tls\") pod \"28f97f04-3cc3-4fa0-99ce-471ed9a389a2\" (UID: \"28f97f04-3cc3-4fa0-99ce-471ed9a389a2\") "
Sep 11 12:46:27.849757 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:27.849734    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a6ff6a-3c11-4036-9d57-636a55aa0ca1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "62a6ff6a-3c11-4036-9d57-636a55aa0ca1" (UID: "62a6ff6a-3c11-4036-9d57-636a55aa0ca1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:46:27.849825 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:27.849760    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28f97f04-3cc3-4fa0-99ce-471ed9a389a2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "28f97f04-3cc3-4fa0-99ce-471ed9a389a2" (UID: "28f97f04-3cc3-4fa0-99ce-471ed9a389a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:46:27.851701 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:27.851674    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28f97f04-3cc3-4fa0-99ce-471ed9a389a2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "28f97f04-3cc3-4fa0-99ce-471ed9a389a2" (UID: "28f97f04-3cc3-4fa0-99ce-471ed9a389a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:46:27.851810 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:27.851698    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a6ff6a-3c11-4036-9d57-636a55aa0ca1-kube-api-access-nlwkg" (OuterVolumeSpecName: "kube-api-access-nlwkg") pod "62a6ff6a-3c11-4036-9d57-636a55aa0ca1" (UID: "62a6ff6a-3c11-4036-9d57-636a55aa0ca1"). InnerVolumeSpecName "kube-api-access-nlwkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:46:27.851810 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:27.851730    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f97f04-3cc3-4fa0-99ce-471ed9a389a2-kube-api-access-mbjbp" (OuterVolumeSpecName: "kube-api-access-mbjbp") pod "28f97f04-3cc3-4fa0-99ce-471ed9a389a2" (UID: "28f97f04-3cc3-4fa0-99ce-471ed9a389a2"). InnerVolumeSpecName "kube-api-access-mbjbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:46:27.851810 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:27.851722    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28f97f04-3cc3-4fa0-99ce-471ed9a389a2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "28f97f04-3cc3-4fa0-99ce-471ed9a389a2" (UID: "28f97f04-3cc3-4fa0-99ce-471ed9a389a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:46:27.950430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:27.950393    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f97f04-3cc3-4fa0-99ce-471ed9a389a2-openshift-service-ca\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:46:27.950430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:27.950428    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f97f04-3cc3-4fa0-99ce-471ed9a389a2-metrics-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:46:27.950430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:27.950438    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbjbp\" (UniqueName: \"kubernetes.io/projected/28f97f04-3cc3-4fa0-99ce-471ed9a389a2-kube-api-access-mbjbp\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:46:27.950628 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:27.950448    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlwkg\" (UniqueName: \"kubernetes.io/projected/62a6ff6a-3c11-4036-9d57-636a55aa0ca1-kube-api-access-nlwkg\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:46:27.950628 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:27.950456    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f97f04-3cc3-4fa0-99ce-471ed9a389a2-maas-api-tls\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:46:27.950628 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:27.950464    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a6ff6a-3c11-4036-9d57-636a55aa0ca1-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:46:28.052530 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:28.052446    2561 generic.go:356] "Generic (PLEG): container finished" podID="62a6ff6a-3c11-4036-9d57-636a55aa0ca1" containerID="1b1cf1d126d557ce970b8dcff3468e5563273c8abd825480bb77cb0ff98a8cfa" exitCode=0
Sep 11 12:46:28.052530 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:28.052513    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv"
Sep 11 12:46:28.052724 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:28.052533    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv" event={"ID":"62a6ff6a-3c11-4036-9d57-636a55aa0ca1","Type":"ContainerDied","Data":"1b1cf1d126d557ce970b8dcff3468e5563273c8abd825480bb77cb0ff98a8cfa"}
Sep 11 12:46:28.052724 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:28.052623    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv" event={"ID":"62a6ff6a-3c11-4036-9d57-636a55aa0ca1","Type":"ContainerDied","Data":"24402314cfefdeeb87a127abc7f4eb6e54081d552e7d2f7c8589843796d61049"}
Sep 11 12:46:28.052724 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:28.052652    2561 scope.go:117] "RemoveContainer" containerID="1b1cf1d126d557ce970b8dcff3468e5563273c8abd825480bb77cb0ff98a8cfa"
Sep 11 12:46:28.054280 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:28.054256    2561 generic.go:356] "Generic (PLEG): container finished" podID="28f97f04-3cc3-4fa0-99ce-471ed9a389a2" containerID="e253aaf8524bb84969349e766af52e61f86a2f73bbc6f3107ba336d825b96d3e" exitCode=0
Sep 11 12:46:28.054406 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:28.054322    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh"
Sep 11 12:46:28.054406 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:28.054342    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh" event={"ID":"28f97f04-3cc3-4fa0-99ce-471ed9a389a2","Type":"ContainerDied","Data":"e253aaf8524bb84969349e766af52e61f86a2f73bbc6f3107ba336d825b96d3e"}
Sep 11 12:46:28.054406 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:28.054393    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh" event={"ID":"28f97f04-3cc3-4fa0-99ce-471ed9a389a2","Type":"ContainerDied","Data":"024c54b783c19aa7b209b15ac4680e63baa82e5519f303c0643820c2c9105f2e"}
Sep 11 12:46:28.061602 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:28.061586    2561 scope.go:117] "RemoveContainer" containerID="1b1cf1d126d557ce970b8dcff3468e5563273c8abd825480bb77cb0ff98a8cfa"
Sep 11 12:46:28.061860 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:46:28.061843    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b1cf1d126d557ce970b8dcff3468e5563273c8abd825480bb77cb0ff98a8cfa\": container with ID starting with 1b1cf1d126d557ce970b8dcff3468e5563273c8abd825480bb77cb0ff98a8cfa not found: ID does not exist" containerID="1b1cf1d126d557ce970b8dcff3468e5563273c8abd825480bb77cb0ff98a8cfa"
Sep 11 12:46:28.061908 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:28.061868    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b1cf1d126d557ce970b8dcff3468e5563273c8abd825480bb77cb0ff98a8cfa"} err="failed to get container status \"1b1cf1d126d557ce970b8dcff3468e5563273c8abd825480bb77cb0ff98a8cfa\": rpc error: code = NotFound desc = could not find container \"1b1cf1d126d557ce970b8dcff3468e5563273c8abd825480bb77cb0ff98a8cfa\": container with ID starting with 1b1cf1d126d557ce970b8dcff3468e5563273c8abd825480bb77cb0ff98a8cfa not found: ID does not exist"
Sep 11 12:46:28.061908 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:28.061885    2561 scope.go:117] "RemoveContainer" containerID="e253aaf8524bb84969349e766af52e61f86a2f73bbc6f3107ba336d825b96d3e"
Sep 11 12:46:28.070211 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:28.070182    2561 scope.go:117] "RemoveContainer" containerID="e253aaf8524bb84969349e766af52e61f86a2f73bbc6f3107ba336d825b96d3e"
Sep 11 12:46:28.070487 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:46:28.070469    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e253aaf8524bb84969349e766af52e61f86a2f73bbc6f3107ba336d825b96d3e\": container with ID starting with e253aaf8524bb84969349e766af52e61f86a2f73bbc6f3107ba336d825b96d3e not found: ID does not exist" containerID="e253aaf8524bb84969349e766af52e61f86a2f73bbc6f3107ba336d825b96d3e"
Sep 11 12:46:28.070545 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:28.070493    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e253aaf8524bb84969349e766af52e61f86a2f73bbc6f3107ba336d825b96d3e"} err="failed to get container status \"e253aaf8524bb84969349e766af52e61f86a2f73bbc6f3107ba336d825b96d3e\": rpc error: code = NotFound desc = could not find container \"e253aaf8524bb84969349e766af52e61f86a2f73bbc6f3107ba336d825b96d3e\": container with ID starting with e253aaf8524bb84969349e766af52e61f86a2f73bbc6f3107ba336d825b96d3e not found: ID does not exist"
Sep 11 12:46:28.081093 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:28.081073    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv"]
Sep 11 12:46:28.085728 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:28.085706    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a1087-7b97b7f87-nq5qv"]
Sep 11 12:46:28.096876 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:28.096852    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh"]
Sep 11 12:46:28.106903 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:28.106881    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a1087-7cbdd456c8-7kswh"]
Sep 11 12:46:28.933871 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:28.933795    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28f97f04-3cc3-4fa0-99ce-471ed9a389a2" path="/var/lib/kubelet/pods/28f97f04-3cc3-4fa0-99ce-471ed9a389a2/volumes"
Sep 11 12:46:28.934206 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:28.934154    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62a6ff6a-3c11-4036-9d57-636a55aa0ca1" path="/var/lib/kubelet/pods/62a6ff6a-3c11-4036-9d57-636a55aa0ca1/volumes"
Sep 11 12:46:29.229049 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:29.229024    2561 scope.go:117] "RemoveContainer" containerID="c5b6b56b342cdc331a90f26577aff9098bb948ef72cf3ed19c2a743947fd5d69"
Sep 11 12:46:29.237382 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:29.237362    2561 scope.go:117] "RemoveContainer" containerID="bc2d0531d2fdb1f3c69d861449255335d8bcbc90dff9e31d74f07b276b891142"
Sep 11 12:46:29.245002 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:29.244981    2561 scope.go:117] "RemoveContainer" containerID="63d7c3dfaac7ef7f0101500bf63fb663e48f2ca6eb96e46b9fd06fc83a170ef7"
Sep 11 12:46:33.088387 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:33.088346    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"]
Sep 11 12:46:36.543838 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:36.543804    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:46:48.107316 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:48.107272    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q" podUID="8c130d3d-73c4-495e-a903-2ebda2a0f3d9" containerName="main" containerID="cri-o://dd702e1e3ac33d81e7adf0eef044c73c8207f2bcfbf6d602c1e56c44edbd9f0f" gracePeriod=45
Sep 11 12:46:48.354279 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:48.354254    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:46:48.426396 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:48.426322    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-tmp-dir\") pod \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\" (UID: \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\") "
Sep 11 12:46:48.426396 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:48.426367    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-home\") pod \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\" (UID: \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\") "
Sep 11 12:46:48.426622 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:48.426406    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-model-cache\") pod \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\" (UID: \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\") "
Sep 11 12:46:48.426622 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:48.426457    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-dshm\") pod \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\" (UID: \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\") "
Sep 11 12:46:48.426622 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:48.426481    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vk7d6\" (UniqueName: \"kubernetes.io/projected/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-kube-api-access-vk7d6\") pod \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\" (UID: \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\") "
Sep 11 12:46:48.426622 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:48.426524    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-tls-certs\") pod \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\" (UID: \"8c130d3d-73c4-495e-a903-2ebda2a0f3d9\") "
Sep 11 12:46:48.426836 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:48.426683    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-home" (OuterVolumeSpecName: "home") pod "8c130d3d-73c4-495e-a903-2ebda2a0f3d9" (UID: "8c130d3d-73c4-495e-a903-2ebda2a0f3d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:46:48.426836 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:48.426714    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8c130d3d-73c4-495e-a903-2ebda2a0f3d9" (UID: "8c130d3d-73c4-495e-a903-2ebda2a0f3d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:46:48.426836 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:48.426714    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "8c130d3d-73c4-495e-a903-2ebda2a0f3d9" (UID: "8c130d3d-73c4-495e-a903-2ebda2a0f3d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:46:48.426991 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:48.426855    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-tmp-dir\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:46:48.426991 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:48.426875    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-home\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:46:48.426991 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:48.426889    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-model-cache\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:46:48.428732 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:48.428706    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-dshm" (OuterVolumeSpecName: "dshm") pod "8c130d3d-73c4-495e-a903-2ebda2a0f3d9" (UID: "8c130d3d-73c4-495e-a903-2ebda2a0f3d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:46:48.428856 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:48.428755    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c130d3d-73c4-495e-a903-2ebda2a0f3d9" (UID: "8c130d3d-73c4-495e-a903-2ebda2a0f3d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:46:48.428856 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:48.428768    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-kube-api-access-vk7d6" (OuterVolumeSpecName: "kube-api-access-vk7d6") pod "8c130d3d-73c4-495e-a903-2ebda2a0f3d9" (UID: "8c130d3d-73c4-495e-a903-2ebda2a0f3d9"). InnerVolumeSpecName "kube-api-access-vk7d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:46:48.527490 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:48.527462    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-dshm\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:46:48.527490 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:48.527488    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk7d6\" (UniqueName: \"kubernetes.io/projected/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-kube-api-access-vk7d6\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:46:48.527650 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:48.527499    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c130d3d-73c4-495e-a903-2ebda2a0f3d9-tls-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:46:49.154654 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:49.154619    2561 generic.go:356] "Generic (PLEG): container finished" podID="8c130d3d-73c4-495e-a903-2ebda2a0f3d9" containerID="dd702e1e3ac33d81e7adf0eef044c73c8207f2bcfbf6d602c1e56c44edbd9f0f" exitCode=0
Sep 11 12:46:49.155030 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:49.154656    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"
Sep 11 12:46:49.155030 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:49.154687    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q" event={"ID":"8c130d3d-73c4-495e-a903-2ebda2a0f3d9","Type":"ContainerDied","Data":"dd702e1e3ac33d81e7adf0eef044c73c8207f2bcfbf6d602c1e56c44edbd9f0f"}
Sep 11 12:46:49.155030 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:49.154724    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q" event={"ID":"8c130d3d-73c4-495e-a903-2ebda2a0f3d9","Type":"ContainerDied","Data":"3859f150d53a3f62da87d1031e9fc459c67c9d0827984a913863235adce1cb41"}
Sep 11 12:46:49.155030 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:49.154743    2561 scope.go:117] "RemoveContainer" containerID="dd702e1e3ac33d81e7adf0eef044c73c8207f2bcfbf6d602c1e56c44edbd9f0f"
Sep 11 12:46:49.162938 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:49.162919    2561 scope.go:117] "RemoveContainer" containerID="dd702e1e3ac33d81e7adf0eef044c73c8207f2bcfbf6d602c1e56c44edbd9f0f"
Sep 11 12:46:49.163224 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:46:49.163203    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd702e1e3ac33d81e7adf0eef044c73c8207f2bcfbf6d602c1e56c44edbd9f0f\": container with ID starting with dd702e1e3ac33d81e7adf0eef044c73c8207f2bcfbf6d602c1e56c44edbd9f0f not found: ID does not exist" containerID="dd702e1e3ac33d81e7adf0eef044c73c8207f2bcfbf6d602c1e56c44edbd9f0f"
Sep 11 12:46:49.163288 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:49.163232    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd702e1e3ac33d81e7adf0eef044c73c8207f2bcfbf6d602c1e56c44edbd9f0f"} err="failed to get container status \"dd702e1e3ac33d81e7adf0eef044c73c8207f2bcfbf6d602c1e56c44edbd9f0f\": rpc error: code = NotFound desc = could not find container \"dd702e1e3ac33d81e7adf0eef044c73c8207f2bcfbf6d602c1e56c44edbd9f0f\": container with ID starting with dd702e1e3ac33d81e7adf0eef044c73c8207f2bcfbf6d602c1e56c44edbd9f0f not found: ID does not exist"
Sep 11 12:46:49.171933 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:49.171910    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"]
Sep 11 12:46:49.178036 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:49.178015    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-0a1087/ipp-route-0a1087-kserve-7b648854ff-sl66q"]
Sep 11 12:46:50.932896 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:46:50.932863    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c130d3d-73c4-495e-a903-2ebda2a0f3d9" path="/var/lib/kubelet/pods/8c130d3d-73c4-495e-a903-2ebda2a0f3d9/volumes"
Sep 11 12:47:29.282892 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:29.282818    2561 scope.go:117] "RemoveContainer" containerID="7dc1b2b76b4ada6232be46e47d2088ce914c5c5d4c7aa700c2bd98b46a192612"
Sep 11 12:47:35.156758 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.156721    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"]
Sep 11 12:47:35.157217 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.156951    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms" podUID="f5128075-e465-439f-ac27-63a19d7f8ec9" containerName="maas-api" containerID="cri-o://22c82194c17f26a1703c3b71ad00652dad29f6018e12f92599205d66e78eb359" gracePeriod=30
Sep 11 12:47:35.321019 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.320978    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn"]
Sep 11 12:47:35.321395 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.321253    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn" podUID="56dbcd44-a271-4389-a6b2-e018fa5e0867" containerName="payload-pre-processing" containerID="cri-o://c025dd8b5a9a24387b450b0bcb2c68b1509bc1d7fececd41919e763b4abfadbe" gracePeriod=30
Sep 11 12:47:35.366929 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.366893    2561 generic.go:356] "Generic (PLEG): container finished" podID="f5128075-e465-439f-ac27-63a19d7f8ec9" containerID="22c82194c17f26a1703c3b71ad00652dad29f6018e12f92599205d66e78eb359" exitCode=0
Sep 11 12:47:35.367077 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.366941    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms" event={"ID":"f5128075-e465-439f-ac27-63a19d7f8ec9","Type":"ContainerDied","Data":"22c82194c17f26a1703c3b71ad00652dad29f6018e12f92599205d66e78eb359"}
Sep 11 12:47:35.445594 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.445570    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"
Sep 11 12:47:35.521634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.521604    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5128075-e465-439f-ac27-63a19d7f8ec9-metrics-certs\") pod \"f5128075-e465-439f-ac27-63a19d7f8ec9\" (UID: \"f5128075-e465-439f-ac27-63a19d7f8ec9\") "
Sep 11 12:47:35.521796 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.521667    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bf5kd\" (UniqueName: \"kubernetes.io/projected/f5128075-e465-439f-ac27-63a19d7f8ec9-kube-api-access-bf5kd\") pod \"f5128075-e465-439f-ac27-63a19d7f8ec9\" (UID: \"f5128075-e465-439f-ac27-63a19d7f8ec9\") "
Sep 11 12:47:35.521796 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.521707    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5128075-e465-439f-ac27-63a19d7f8ec9-maas-api-tls\") pod \"f5128075-e465-439f-ac27-63a19d7f8ec9\" (UID: \"f5128075-e465-439f-ac27-63a19d7f8ec9\") "
Sep 11 12:47:35.521904 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.521796    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5128075-e465-439f-ac27-63a19d7f8ec9-openshift-service-ca\") pod \"f5128075-e465-439f-ac27-63a19d7f8ec9\" (UID: \"f5128075-e465-439f-ac27-63a19d7f8ec9\") "
Sep 11 12:47:35.522405 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.522356    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5128075-e465-439f-ac27-63a19d7f8ec9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f5128075-e465-439f-ac27-63a19d7f8ec9" (UID: "f5128075-e465-439f-ac27-63a19d7f8ec9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:47:35.524656 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.524625    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5128075-e465-439f-ac27-63a19d7f8ec9-kube-api-access-bf5kd" (OuterVolumeSpecName: "kube-api-access-bf5kd") pod "f5128075-e465-439f-ac27-63a19d7f8ec9" (UID: "f5128075-e465-439f-ac27-63a19d7f8ec9"). InnerVolumeSpecName "kube-api-access-bf5kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:47:35.524772 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.524740    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5128075-e465-439f-ac27-63a19d7f8ec9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f5128075-e465-439f-ac27-63a19d7f8ec9" (UID: "f5128075-e465-439f-ac27-63a19d7f8ec9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:47:35.524772 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.524752    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5128075-e465-439f-ac27-63a19d7f8ec9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f5128075-e465-439f-ac27-63a19d7f8ec9" (UID: "f5128075-e465-439f-ac27-63a19d7f8ec9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:47:35.550912 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.550890    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn"
Sep 11 12:47:35.622953 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.622927    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzvkz\" (UniqueName: \"kubernetes.io/projected/56dbcd44-a271-4389-a6b2-e018fa5e0867-kube-api-access-rzvkz\") pod \"56dbcd44-a271-4389-a6b2-e018fa5e0867\" (UID: \"56dbcd44-a271-4389-a6b2-e018fa5e0867\") "
Sep 11 12:47:35.622953 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.622959    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56dbcd44-a271-4389-a6b2-e018fa5e0867-plugins-config-volume\") pod \"56dbcd44-a271-4389-a6b2-e018fa5e0867\" (UID: \"56dbcd44-a271-4389-a6b2-e018fa5e0867\") "
Sep 11 12:47:35.623135 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.623111    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5128075-e465-439f-ac27-63a19d7f8ec9-maas-api-tls\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:47:35.623135 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.623121    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5128075-e465-439f-ac27-63a19d7f8ec9-openshift-service-ca\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:47:35.623135 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.623131    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5128075-e465-439f-ac27-63a19d7f8ec9-metrics-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:47:35.623259 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.623139    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf5kd\" (UniqueName: \"kubernetes.io/projected/f5128075-e465-439f-ac27-63a19d7f8ec9-kube-api-access-bf5kd\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:47:35.623386 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.623360    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56dbcd44-a271-4389-a6b2-e018fa5e0867-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56dbcd44-a271-4389-a6b2-e018fa5e0867" (UID: "56dbcd44-a271-4389-a6b2-e018fa5e0867"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:47:35.624950 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.624930    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56dbcd44-a271-4389-a6b2-e018fa5e0867-kube-api-access-rzvkz" (OuterVolumeSpecName: "kube-api-access-rzvkz") pod "56dbcd44-a271-4389-a6b2-e018fa5e0867" (UID: "56dbcd44-a271-4389-a6b2-e018fa5e0867"). InnerVolumeSpecName "kube-api-access-rzvkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:47:35.724557 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.724485    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzvkz\" (UniqueName: \"kubernetes.io/projected/56dbcd44-a271-4389-a6b2-e018fa5e0867-kube-api-access-rzvkz\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:47:35.724557 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:35.724511    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56dbcd44-a271-4389-a6b2-e018fa5e0867-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:47:36.373007 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:36.372965    2561 generic.go:356] "Generic (PLEG): container finished" podID="56dbcd44-a271-4389-a6b2-e018fa5e0867" containerID="c025dd8b5a9a24387b450b0bcb2c68b1509bc1d7fececd41919e763b4abfadbe" exitCode=0
Sep 11 12:47:36.373496 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:36.373011    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn"
Sep 11 12:47:36.373496 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:36.373045    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn" event={"ID":"56dbcd44-a271-4389-a6b2-e018fa5e0867","Type":"ContainerDied","Data":"c025dd8b5a9a24387b450b0bcb2c68b1509bc1d7fececd41919e763b4abfadbe"}
Sep 11 12:47:36.373496 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:36.373096    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn" event={"ID":"56dbcd44-a271-4389-a6b2-e018fa5e0867","Type":"ContainerDied","Data":"d229c6908a909d2e32c1e284477994d58bcde4a9365f6336f30256097c58cf4f"}
Sep 11 12:47:36.373496 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:36.373127    2561 scope.go:117] "RemoveContainer" containerID="c025dd8b5a9a24387b450b0bcb2c68b1509bc1d7fececd41919e763b4abfadbe"
Sep 11 12:47:36.374617 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:36.374599    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms" event={"ID":"f5128075-e465-439f-ac27-63a19d7f8ec9","Type":"ContainerDied","Data":"b34b6c08a1aa6509fc8b19d3f8662749eb80a082a1d5882e2a25b115139cc3aa"}
Sep 11 12:47:36.374617 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:36.374610    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"
Sep 11 12:47:36.382580 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:36.382546    2561 scope.go:117] "RemoveContainer" containerID="c025dd8b5a9a24387b450b0bcb2c68b1509bc1d7fececd41919e763b4abfadbe"
Sep 11 12:47:36.382856 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:47:36.382832    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c025dd8b5a9a24387b450b0bcb2c68b1509bc1d7fececd41919e763b4abfadbe\": container with ID starting with c025dd8b5a9a24387b450b0bcb2c68b1509bc1d7fececd41919e763b4abfadbe not found: ID does not exist" containerID="c025dd8b5a9a24387b450b0bcb2c68b1509bc1d7fececd41919e763b4abfadbe"
Sep 11 12:47:36.382949 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:36.382862    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c025dd8b5a9a24387b450b0bcb2c68b1509bc1d7fececd41919e763b4abfadbe"} err="failed to get container status \"c025dd8b5a9a24387b450b0bcb2c68b1509bc1d7fececd41919e763b4abfadbe\": rpc error: code = NotFound desc = could not find container \"c025dd8b5a9a24387b450b0bcb2c68b1509bc1d7fececd41919e763b4abfadbe\": container with ID starting with c025dd8b5a9a24387b450b0bcb2c68b1509bc1d7fececd41919e763b4abfadbe not found: ID does not exist"
Sep 11 12:47:36.382949 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:36.382877    2561 scope.go:117] "RemoveContainer" containerID="22c82194c17f26a1703c3b71ad00652dad29f6018e12f92599205d66e78eb359"
Sep 11 12:47:36.401992 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:36.401972    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"]
Sep 11 12:47:36.412199 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:36.412170    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5ae2a-54fcf5ff54-b2tms"]
Sep 11 12:47:36.425807 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:36.425761    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn"]
Sep 11 12:47:36.431126 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:36.431106    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d5ae2a-798dcb9fc8-ntjkn"]
Sep 11 12:47:36.932341 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:36.932307    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56dbcd44-a271-4389-a6b2-e018fa5e0867" path="/var/lib/kubelet/pods/56dbcd44-a271-4389-a6b2-e018fa5e0867/volumes"
Sep 11 12:47:36.932728 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:36.932715    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5128075-e465-439f-ac27-63a19d7f8ec9" path="/var/lib/kubelet/pods/f5128075-e465-439f-ac27-63a19d7f8ec9/volumes"
Sep 11 12:47:38.560200 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:38.560151    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:47:44.146179 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.146145    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa824-7kfr7"]
Sep 11 12:47:44.146672 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.146529    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18320f23-b4a4-491e-bb68-27c3983eca0f" containerName="payload-pre-processing"
Sep 11 12:47:44.146672 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.146545    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="18320f23-b4a4-491e-bb68-27c3983eca0f" containerName="payload-pre-processing"
Sep 11 12:47:44.146672 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.146565    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a6ff6a-3c11-4036-9d57-636a55aa0ca1" containerName="payload-processing"
Sep 11 12:47:44.146672 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.146571    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a6ff6a-3c11-4036-9d57-636a55aa0ca1" containerName="payload-processing"
Sep 11 12:47:44.146672 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.146578    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56dbcd44-a271-4389-a6b2-e018fa5e0867" containerName="payload-pre-processing"
Sep 11 12:47:44.146672 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.146584    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="56dbcd44-a271-4389-a6b2-e018fa5e0867" containerName="payload-pre-processing"
Sep 11 12:47:44.146672 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.146591    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28f97f04-3cc3-4fa0-99ce-471ed9a389a2" containerName="maas-api"
Sep 11 12:47:44.146672 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.146596    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="28f97f04-3cc3-4fa0-99ce-471ed9a389a2" containerName="maas-api"
Sep 11 12:47:44.146672 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.146601    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="999e3562-6f74-4e8b-b40e-c863102a3d92" containerName="maas-api"
Sep 11 12:47:44.146672 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.146606    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="999e3562-6f74-4e8b-b40e-c863102a3d92" containerName="maas-api"
Sep 11 12:47:44.146672 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.146616    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c130d3d-73c4-495e-a903-2ebda2a0f3d9" containerName="main"
Sep 11 12:47:44.146672 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.146621    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c130d3d-73c4-495e-a903-2ebda2a0f3d9" containerName="main"
Sep 11 12:47:44.146672 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.146631    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5128075-e465-439f-ac27-63a19d7f8ec9" containerName="maas-api"
Sep 11 12:47:44.146672 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.146636    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5128075-e465-439f-ac27-63a19d7f8ec9" containerName="maas-api"
Sep 11 12:47:44.147101 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.146688    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c130d3d-73c4-495e-a903-2ebda2a0f3d9" containerName="main"
Sep 11 12:47:44.147101 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.146694    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="999e3562-6f74-4e8b-b40e-c863102a3d92" containerName="maas-api"
Sep 11 12:47:44.147101 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.146701    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5128075-e465-439f-ac27-63a19d7f8ec9" containerName="maas-api"
Sep 11 12:47:44.147101 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.146708    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="18320f23-b4a4-491e-bb68-27c3983eca0f" containerName="payload-pre-processing"
Sep 11 12:47:44.147101 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.146714    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="56dbcd44-a271-4389-a6b2-e018fa5e0867" containerName="payload-pre-processing"
Sep 11 12:47:44.147101 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.146720    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="28f97f04-3cc3-4fa0-99ce-471ed9a389a2" containerName="maas-api"
Sep 11 12:47:44.147101 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.146726    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="62a6ff6a-3c11-4036-9d57-636a55aa0ca1" containerName="payload-processing"
Sep 11 12:47:44.150933 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.150916    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa824-7kfr7"
Sep 11 12:47:44.154403 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.154348    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 12:47:44.154680 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.154664    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\""
Sep 11 12:47:44.159632 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.159610    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa824-7kfr7"]
Sep 11 12:47:44.300128 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.300088    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb12a6c3-b466-45fc-a6a6-64b817286064-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-aaa824-7kfr7\" (UID: \"eb12a6c3-b466-45fc-a6a6-64b817286064\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa824-7kfr7"
Sep 11 12:47:44.401032 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.400952    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb12a6c3-b466-45fc-a6a6-64b817286064-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-aaa824-7kfr7\" (UID: \"eb12a6c3-b466-45fc-a6a6-64b817286064\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa824-7kfr7"
Sep 11 12:47:44.401588 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.401566    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb12a6c3-b466-45fc-a6a6-64b817286064-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-aaa824-7kfr7\" (UID: \"eb12a6c3-b466-45fc-a6a6-64b817286064\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa824-7kfr7"
Sep 11 12:47:44.462582 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.462546    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa824-7kfr7"
Sep 11 12:47:44.746250 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:44.746218    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa824-7kfr7"]
Sep 11 12:47:44.749338 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:47:44.749304    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb12a6c3_b466_45fc_a6a6_64b817286064.slice/crio-e905b5e8d03676222528e3d98b3ee512461dd13e33c68a0a55d2d71ca3c2990b WatchSource:0}: Error finding container e905b5e8d03676222528e3d98b3ee512461dd13e33c68a0a55d2d71ca3c2990b: Status 404 returned error can't find the container with id e905b5e8d03676222528e3d98b3ee512461dd13e33c68a0a55d2d71ca3c2990b
Sep 11 12:47:45.415884 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:45.415854    2561 generic.go:356] "Generic (PLEG): container finished" podID="eb12a6c3-b466-45fc-a6a6-64b817286064" containerID="434cc1e3192c4c56ccfdf612252424ca1bf653fe80bc4c18ff1772c579d386ac" exitCode=0
Sep 11 12:47:45.416167 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:45.415911    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa824-7kfr7" event={"ID":"eb12a6c3-b466-45fc-a6a6-64b817286064","Type":"ContainerDied","Data":"434cc1e3192c4c56ccfdf612252424ca1bf653fe80bc4c18ff1772c579d386ac"}
Sep 11 12:47:45.416167 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:45.415977    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa824-7kfr7" event={"ID":"eb12a6c3-b466-45fc-a6a6-64b817286064","Type":"ContainerStarted","Data":"e905b5e8d03676222528e3d98b3ee512461dd13e33c68a0a55d2d71ca3c2990b"}
Sep 11 12:47:46.545741 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:46.545719    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa824-7kfr7"
Sep 11 12:47:46.720079 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:46.720001    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb12a6c3-b466-45fc-a6a6-64b817286064-maas-api-service-ca\") pod \"eb12a6c3-b466-45fc-a6a6-64b817286064\" (UID: \"eb12a6c3-b466-45fc-a6a6-64b817286064\") "
Sep 11 12:47:46.720378 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:46.720352    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb12a6c3-b466-45fc-a6a6-64b817286064-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb12a6c3-b466-45fc-a6a6-64b817286064" (UID: "eb12a6c3-b466-45fc-a6a6-64b817286064"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:47:46.821532 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:46.821502    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb12a6c3-b466-45fc-a6a6-64b817286064-maas-api-service-ca\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:47:47.427396 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:47.427356    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa824-7kfr7" event={"ID":"eb12a6c3-b466-45fc-a6a6-64b817286064","Type":"ContainerDied","Data":"e905b5e8d03676222528e3d98b3ee512461dd13e33c68a0a55d2d71ca3c2990b"}
Sep 11 12:47:47.427396 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:47.427378    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa824-7kfr7"
Sep 11 12:47:47.427651 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:47.427407    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e905b5e8d03676222528e3d98b3ee512461dd13e33c68a0a55d2d71ca3c2990b"
Sep 11 12:47:54.645241 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:54.645200    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa824-29818845-wfq7x"]
Sep 11 12:47:54.649148 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:54.649120    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa824-29818845-wfq7x"]
Sep 11 12:47:54.678274 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:54.678248    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd"]
Sep 11 12:47:54.678698 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:54.678643    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd" podUID="7e05c25a-76ac-4afd-9472-d8219614eb37" containerName="payload-pre-processing" containerID="cri-o://f10a4afeae1dcd58d5d1e358aef498a6b44c285ae15972e0834b3449fd5a491b" gracePeriod=30
Sep 11 12:47:54.843449 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:54.843411    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa824-7kfr7"]
Sep 11 12:47:54.846137 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:54.846107    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aaa824-7kfr7"]
Sep 11 12:47:54.922709 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:54.922684    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd"
Sep 11 12:47:54.931242 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:54.931213    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6f1aff2-550b-4dd2-bbe3-7acdd08acf76" path="/var/lib/kubelet/pods/b6f1aff2-550b-4dd2-bbe3-7acdd08acf76/volumes"
Sep 11 12:47:54.931563 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:54.931549    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb12a6c3-b466-45fc-a6a6-64b817286064" path="/var/lib/kubelet/pods/eb12a6c3-b466-45fc-a6a6-64b817286064/volumes"
Sep 11 12:47:55.092282 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:55.092252    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e05c25a-76ac-4afd-9472-d8219614eb37-plugins-config-volume\") pod \"7e05c25a-76ac-4afd-9472-d8219614eb37\" (UID: \"7e05c25a-76ac-4afd-9472-d8219614eb37\") "
Sep 11 12:47:55.092473 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:55.092331    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppmm6\" (UniqueName: \"kubernetes.io/projected/7e05c25a-76ac-4afd-9472-d8219614eb37-kube-api-access-ppmm6\") pod \"7e05c25a-76ac-4afd-9472-d8219614eb37\" (UID: \"7e05c25a-76ac-4afd-9472-d8219614eb37\") "
Sep 11 12:47:55.092623 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:55.092598    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e05c25a-76ac-4afd-9472-d8219614eb37-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7e05c25a-76ac-4afd-9472-d8219614eb37" (UID: "7e05c25a-76ac-4afd-9472-d8219614eb37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:47:55.094255 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:55.094232    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e05c25a-76ac-4afd-9472-d8219614eb37-kube-api-access-ppmm6" (OuterVolumeSpecName: "kube-api-access-ppmm6") pod "7e05c25a-76ac-4afd-9472-d8219614eb37" (UID: "7e05c25a-76ac-4afd-9472-d8219614eb37"). InnerVolumeSpecName "kube-api-access-ppmm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:47:55.193760 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:55.193686    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e05c25a-76ac-4afd-9472-d8219614eb37-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:47:55.193760 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:55.193715    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppmm6\" (UniqueName: \"kubernetes.io/projected/7e05c25a-76ac-4afd-9472-d8219614eb37-kube-api-access-ppmm6\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:47:55.464577 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:55.464492    2561 generic.go:356] "Generic (PLEG): container finished" podID="7e05c25a-76ac-4afd-9472-d8219614eb37" containerID="f10a4afeae1dcd58d5d1e358aef498a6b44c285ae15972e0834b3449fd5a491b" exitCode=0
Sep 11 12:47:55.464577 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:55.464538    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd"
Sep 11 12:47:55.464754 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:55.464579    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd" event={"ID":"7e05c25a-76ac-4afd-9472-d8219614eb37","Type":"ContainerDied","Data":"f10a4afeae1dcd58d5d1e358aef498a6b44c285ae15972e0834b3449fd5a491b"}
Sep 11 12:47:55.464754 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:55.464634    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd" event={"ID":"7e05c25a-76ac-4afd-9472-d8219614eb37","Type":"ContainerDied","Data":"fc4c2b75c5345bb7479a23ebf823d6ffb2c26c356d45843e795a6409700f7976"}
Sep 11 12:47:55.464754 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:55.464658    2561 scope.go:117] "RemoveContainer" containerID="f10a4afeae1dcd58d5d1e358aef498a6b44c285ae15972e0834b3449fd5a491b"
Sep 11 12:47:55.473257 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:55.473240    2561 scope.go:117] "RemoveContainer" containerID="f10a4afeae1dcd58d5d1e358aef498a6b44c285ae15972e0834b3449fd5a491b"
Sep 11 12:47:55.473506 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:47:55.473488    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f10a4afeae1dcd58d5d1e358aef498a6b44c285ae15972e0834b3449fd5a491b\": container with ID starting with f10a4afeae1dcd58d5d1e358aef498a6b44c285ae15972e0834b3449fd5a491b not found: ID does not exist" containerID="f10a4afeae1dcd58d5d1e358aef498a6b44c285ae15972e0834b3449fd5a491b"
Sep 11 12:47:55.473561 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:55.473513    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f10a4afeae1dcd58d5d1e358aef498a6b44c285ae15972e0834b3449fd5a491b"} err="failed to get container status \"f10a4afeae1dcd58d5d1e358aef498a6b44c285ae15972e0834b3449fd5a491b\": rpc error: code = NotFound desc = could not find container \"f10a4afeae1dcd58d5d1e358aef498a6b44c285ae15972e0834b3449fd5a491b\": container with ID starting with f10a4afeae1dcd58d5d1e358aef498a6b44c285ae15972e0834b3449fd5a491b not found: ID does not exist"
Sep 11 12:47:55.485897 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:55.485872    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd"]
Sep 11 12:47:55.489949 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:55.489927    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-aaa824-647ccdf66-ghssd"]
Sep 11 12:47:56.932315 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:56.932279    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e05c25a-76ac-4afd-9472-d8219614eb37" path="/var/lib/kubelet/pods/7e05c25a-76ac-4afd-9472-d8219614eb37/volumes"
Sep 11 12:47:57.357082 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:47:57.357042    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:48:02.852970 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:02.852928    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4502d-5fm24"]
Sep 11 12:48:02.853459 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:02.853438    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e05c25a-76ac-4afd-9472-d8219614eb37" containerName="payload-pre-processing"
Sep 11 12:48:02.853529 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:02.853462    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e05c25a-76ac-4afd-9472-d8219614eb37" containerName="payload-pre-processing"
Sep 11 12:48:02.853529 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:02.853494    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb12a6c3-b466-45fc-a6a6-64b817286064" containerName="revoke-keys"
Sep 11 12:48:02.853529 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:02.853504    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb12a6c3-b466-45fc-a6a6-64b817286064" containerName="revoke-keys"
Sep 11 12:48:02.853695 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:02.853590    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e05c25a-76ac-4afd-9472-d8219614eb37" containerName="payload-pre-processing"
Sep 11 12:48:02.853695 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:02.853603    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb12a6c3-b466-45fc-a6a6-64b817286064" containerName="revoke-keys"
Sep 11 12:48:02.856498 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:02.856476    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4502d-5fm24"
Sep 11 12:48:02.859044 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:02.859025    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\""
Sep 11 12:48:02.859475 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:02.859457    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 12:48:02.866462 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:02.866439    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4502d-5fm24"]
Sep 11 12:48:02.961867 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:02.961838    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f0c089-2419-415f-b178-15da6538b7f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f4502d-5fm24\" (UID: \"c2f0c089-2419-415f-b178-15da6538b7f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4502d-5fm24"
Sep 11 12:48:03.062978 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:03.062951    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f0c089-2419-415f-b178-15da6538b7f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f4502d-5fm24\" (UID: \"c2f0c089-2419-415f-b178-15da6538b7f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4502d-5fm24"
Sep 11 12:48:03.063588 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:03.063569    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f0c089-2419-415f-b178-15da6538b7f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-f4502d-5fm24\" (UID: \"c2f0c089-2419-415f-b178-15da6538b7f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4502d-5fm24"
Sep 11 12:48:03.167270 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:03.167207    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4502d-5fm24"
Sep 11 12:48:03.452585 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:03.452560    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4502d-5fm24"]
Sep 11 12:48:03.454619 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:48:03.454591    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f0c089_2419_415f_b178_15da6538b7f0.slice/crio-383284e9cf479bc430ee45fd730353bbeb206b82eb8f0bd00496eb2d80bd3230 WatchSource:0}: Error finding container 383284e9cf479bc430ee45fd730353bbeb206b82eb8f0bd00496eb2d80bd3230: Status 404 returned error can't find the container with id 383284e9cf479bc430ee45fd730353bbeb206b82eb8f0bd00496eb2d80bd3230
Sep 11 12:48:03.503577 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:03.503553    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4502d-5fm24" event={"ID":"c2f0c089-2419-415f-b178-15da6538b7f0","Type":"ContainerStarted","Data":"383284e9cf479bc430ee45fd730353bbeb206b82eb8f0bd00496eb2d80bd3230"}
Sep 11 12:48:04.509606 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:04.509563    2561 generic.go:356] "Generic (PLEG): container finished" podID="c2f0c089-2419-415f-b178-15da6538b7f0" containerID="782a1dcaf9972d8639a5afdf37738149b8422c9b54feedffcd8ae5d5f0e0d7c7" exitCode=0
Sep 11 12:48:04.510018 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:04.509647    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4502d-5fm24" event={"ID":"c2f0c089-2419-415f-b178-15da6538b7f0","Type":"ContainerDied","Data":"782a1dcaf9972d8639a5afdf37738149b8422c9b54feedffcd8ae5d5f0e0d7c7"}
Sep 11 12:48:05.644033 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:05.644009    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4502d-5fm24"
Sep 11 12:48:05.688787 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:05.688759    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f0c089-2419-415f-b178-15da6538b7f0-maas-api-service-ca\") pod \"c2f0c089-2419-415f-b178-15da6538b7f0\" (UID: \"c2f0c089-2419-415f-b178-15da6538b7f0\") "
Sep 11 12:48:05.689067 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:05.689046    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2f0c089-2419-415f-b178-15da6538b7f0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c2f0c089-2419-415f-b178-15da6538b7f0" (UID: "c2f0c089-2419-415f-b178-15da6538b7f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:48:05.790467 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:05.790405    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f0c089-2419-415f-b178-15da6538b7f0-maas-api-service-ca\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:48:06.520045 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:06.520008    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4502d-5fm24" event={"ID":"c2f0c089-2419-415f-b178-15da6538b7f0","Type":"ContainerDied","Data":"383284e9cf479bc430ee45fd730353bbeb206b82eb8f0bd00496eb2d80bd3230"}
Sep 11 12:48:06.520238 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:06.520058    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="383284e9cf479bc430ee45fd730353bbeb206b82eb8f0bd00496eb2d80bd3230"
Sep 11 12:48:06.520238 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:06.520019    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4502d-5fm24"
Sep 11 12:48:13.647086 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:13.647054    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n"]
Sep 11 12:48:13.647476 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:13.647262    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n" podUID="260e8e9e-7393-4be6-b6e7-57a5a1512f40" containerName="payload-pre-processing" containerID="cri-o://79f8ba0480c196dea91ce66c8a6eb6bfa8281e63591a6932f3e64be13584ce01" gracePeriod=30
Sep 11 12:48:13.895891 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:13.895866    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n"
Sep 11 12:48:13.957563 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:13.957493    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260e8e9e-7393-4be6-b6e7-57a5a1512f40-plugins-config-volume\") pod \"260e8e9e-7393-4be6-b6e7-57a5a1512f40\" (UID: \"260e8e9e-7393-4be6-b6e7-57a5a1512f40\") "
Sep 11 12:48:13.957724 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:13.957561    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jd7k\" (UniqueName: \"kubernetes.io/projected/260e8e9e-7393-4be6-b6e7-57a5a1512f40-kube-api-access-8jd7k\") pod \"260e8e9e-7393-4be6-b6e7-57a5a1512f40\" (UID: \"260e8e9e-7393-4be6-b6e7-57a5a1512f40\") "
Sep 11 12:48:13.957913 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:13.957889    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/260e8e9e-7393-4be6-b6e7-57a5a1512f40-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "260e8e9e-7393-4be6-b6e7-57a5a1512f40" (UID: "260e8e9e-7393-4be6-b6e7-57a5a1512f40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:48:13.959681 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:13.959657    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260e8e9e-7393-4be6-b6e7-57a5a1512f40-kube-api-access-8jd7k" (OuterVolumeSpecName: "kube-api-access-8jd7k") pod "260e8e9e-7393-4be6-b6e7-57a5a1512f40" (UID: "260e8e9e-7393-4be6-b6e7-57a5a1512f40"). InnerVolumeSpecName "kube-api-access-8jd7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:48:13.996271 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:13.996242    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4502d-5fm24"]
Sep 11 12:48:13.999427 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:13.999408    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-f4502d-5fm24"]
Sep 11 12:48:14.058615 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:14.058594    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jd7k\" (UniqueName: \"kubernetes.io/projected/260e8e9e-7393-4be6-b6e7-57a5a1512f40-kube-api-access-8jd7k\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:48:14.058716 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:14.058616    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260e8e9e-7393-4be6-b6e7-57a5a1512f40-plugins-config-volume\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:48:14.556026 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:14.555992    2561 generic.go:356] "Generic (PLEG): container finished" podID="260e8e9e-7393-4be6-b6e7-57a5a1512f40" containerID="79f8ba0480c196dea91ce66c8a6eb6bfa8281e63591a6932f3e64be13584ce01" exitCode=0
Sep 11 12:48:14.556247 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:14.556035    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n"
Sep 11 12:48:14.556247 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:14.556051    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n" event={"ID":"260e8e9e-7393-4be6-b6e7-57a5a1512f40","Type":"ContainerDied","Data":"79f8ba0480c196dea91ce66c8a6eb6bfa8281e63591a6932f3e64be13584ce01"}
Sep 11 12:48:14.556247 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:14.556106    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n" event={"ID":"260e8e9e-7393-4be6-b6e7-57a5a1512f40","Type":"ContainerDied","Data":"759f7353f1f451d84aa22c16cb318b9f25c12584ceef6a79979013a55efbd1cd"}
Sep 11 12:48:14.556247 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:14.556127    2561 scope.go:117] "RemoveContainer" containerID="79f8ba0480c196dea91ce66c8a6eb6bfa8281e63591a6932f3e64be13584ce01"
Sep 11 12:48:14.564504 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:14.564488    2561 scope.go:117] "RemoveContainer" containerID="79f8ba0480c196dea91ce66c8a6eb6bfa8281e63591a6932f3e64be13584ce01"
Sep 11 12:48:14.564763 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:48:14.564743    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79f8ba0480c196dea91ce66c8a6eb6bfa8281e63591a6932f3e64be13584ce01\": container with ID starting with 79f8ba0480c196dea91ce66c8a6eb6bfa8281e63591a6932f3e64be13584ce01 not found: ID does not exist" containerID="79f8ba0480c196dea91ce66c8a6eb6bfa8281e63591a6932f3e64be13584ce01"
Sep 11 12:48:14.564810 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:14.564773    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f8ba0480c196dea91ce66c8a6eb6bfa8281e63591a6932f3e64be13584ce01"} err="failed to get container status \"79f8ba0480c196dea91ce66c8a6eb6bfa8281e63591a6932f3e64be13584ce01\": rpc error: code = NotFound desc = could not find container \"79f8ba0480c196dea91ce66c8a6eb6bfa8281e63591a6932f3e64be13584ce01\": container with ID starting with 79f8ba0480c196dea91ce66c8a6eb6bfa8281e63591a6932f3e64be13584ce01 not found: ID does not exist"
Sep 11 12:48:14.592077 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:14.592055    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n"]
Sep 11 12:48:14.596662 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:14.596643    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f4502d-c9f7cb84b-5lt5n"]
Sep 11 12:48:14.931840 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:14.931768    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="260e8e9e-7393-4be6-b6e7-57a5a1512f40" path="/var/lib/kubelet/pods/260e8e9e-7393-4be6-b6e7-57a5a1512f40/volumes"
Sep 11 12:48:14.932272 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:14.932083    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2f0c089-2419-415f-b178-15da6538b7f0" path="/var/lib/kubelet/pods/c2f0c089-2419-415f-b178-15da6538b7f0/volumes"
Sep 11 12:48:15.952799 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:15.952767    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:48:25.027903 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:48:25.027865    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:49:29.013910 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:49:29.013879    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcfm6_88402687-4e5b-411c-be7a-4cb955f90425/ovn-acl-logging/0.log"
Sep 11 12:49:29.020620 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:49:29.020597    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcfm6_88402687-4e5b-411c-be7a-4cb955f90425/ovn-acl-logging/0.log"
Sep 11 12:49:45.337805 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:49:45.337768    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:50:10.350506 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:50:10.350467    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:50:17.956312 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:50:17.956277    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:50:24.688151 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:50:24.688114    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:50:39.036492 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:50:39.036451    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:50:44.636894 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:50:44.636856    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:51:29.338696 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:51:29.338666    2561 scope.go:117] "RemoveContainer" containerID="391c7aaaaaf103be7801e5a950af39b92f3783485272ab161a3ac53e74c2e77b"
Sep 11 12:52:09.629475 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:09.629394    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:52:17.235956 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:17.235922    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:52:24.226318 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:24.226279    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:52:35.403228 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.403181    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"]
Sep 11 12:52:35.403586 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.403575    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2f0c089-2419-415f-b178-15da6538b7f0" containerName="revoke-keys"
Sep 11 12:52:35.403586 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.403587    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2f0c089-2419-415f-b178-15da6538b7f0" containerName="revoke-keys"
Sep 11 12:52:35.403660 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.403595    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="260e8e9e-7393-4be6-b6e7-57a5a1512f40" containerName="payload-pre-processing"
Sep 11 12:52:35.403660 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.403600    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="260e8e9e-7393-4be6-b6e7-57a5a1512f40" containerName="payload-pre-processing"
Sep 11 12:52:35.403733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.403664    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="260e8e9e-7393-4be6-b6e7-57a5a1512f40" containerName="payload-pre-processing"
Sep 11 12:52:35.403733 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.403675    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2f0c089-2419-415f-b178-15da6538b7f0" containerName="revoke-keys"
Sep 11 12:52:35.406644 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.406621    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:52:35.409328 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.409306    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-614772b9-kserve-self-signed-certs\""
Sep 11 12:52:35.416011 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.415988    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"]
Sep 11 12:52:35.540066 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.540037    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23d87d5a-27f0-4374-bd2d-66aa4852939c-tmp-dir\") pod \"e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt\" (UID: \"23d87d5a-27f0-4374-bd2d-66aa4852939c\") " pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:52:35.540270 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.540082    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23d87d5a-27f0-4374-bd2d-66aa4852939c-tls-certs\") pod \"e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt\" (UID: \"23d87d5a-27f0-4374-bd2d-66aa4852939c\") " pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:52:35.540270 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.540100    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23d87d5a-27f0-4374-bd2d-66aa4852939c-home\") pod \"e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt\" (UID: \"23d87d5a-27f0-4374-bd2d-66aa4852939c\") " pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:52:35.540270 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.540118    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23d87d5a-27f0-4374-bd2d-66aa4852939c-model-cache\") pod \"e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt\" (UID: \"23d87d5a-27f0-4374-bd2d-66aa4852939c\") " pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:52:35.540270 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.540221    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqgt5\" (UniqueName: \"kubernetes.io/projected/23d87d5a-27f0-4374-bd2d-66aa4852939c-kube-api-access-hqgt5\") pod \"e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt\" (UID: \"23d87d5a-27f0-4374-bd2d-66aa4852939c\") " pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:52:35.540270 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.540267    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23d87d5a-27f0-4374-bd2d-66aa4852939c-dshm\") pod \"e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt\" (UID: \"23d87d5a-27f0-4374-bd2d-66aa4852939c\") " pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:52:35.641339 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.641303    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23d87d5a-27f0-4374-bd2d-66aa4852939c-tmp-dir\") pod \"e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt\" (UID: \"23d87d5a-27f0-4374-bd2d-66aa4852939c\") " pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:52:35.641493 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.641361    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23d87d5a-27f0-4374-bd2d-66aa4852939c-tls-certs\") pod \"e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt\" (UID: \"23d87d5a-27f0-4374-bd2d-66aa4852939c\") " pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:52:35.641493 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.641385    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23d87d5a-27f0-4374-bd2d-66aa4852939c-home\") pod \"e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt\" (UID: \"23d87d5a-27f0-4374-bd2d-66aa4852939c\") " pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:52:35.641493 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.641407    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23d87d5a-27f0-4374-bd2d-66aa4852939c-model-cache\") pod \"e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt\" (UID: \"23d87d5a-27f0-4374-bd2d-66aa4852939c\") " pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:52:35.641493 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.641440    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqgt5\" (UniqueName: \"kubernetes.io/projected/23d87d5a-27f0-4374-bd2d-66aa4852939c-kube-api-access-hqgt5\") pod \"e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt\" (UID: \"23d87d5a-27f0-4374-bd2d-66aa4852939c\") " pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:52:35.641493 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.641468    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23d87d5a-27f0-4374-bd2d-66aa4852939c-dshm\") pod \"e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt\" (UID: \"23d87d5a-27f0-4374-bd2d-66aa4852939c\") " pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:52:35.641778 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.641754    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23d87d5a-27f0-4374-bd2d-66aa4852939c-tmp-dir\") pod \"e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt\" (UID: \"23d87d5a-27f0-4374-bd2d-66aa4852939c\") " pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:52:35.641778 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.641772    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23d87d5a-27f0-4374-bd2d-66aa4852939c-home\") pod \"e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt\" (UID: \"23d87d5a-27f0-4374-bd2d-66aa4852939c\") " pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:52:35.641947 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.641828    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23d87d5a-27f0-4374-bd2d-66aa4852939c-model-cache\") pod \"e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt\" (UID: \"23d87d5a-27f0-4374-bd2d-66aa4852939c\") " pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:52:35.643690 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.643660    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23d87d5a-27f0-4374-bd2d-66aa4852939c-dshm\") pod \"e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt\" (UID: \"23d87d5a-27f0-4374-bd2d-66aa4852939c\") " pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:52:35.643817 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.643800    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23d87d5a-27f0-4374-bd2d-66aa4852939c-tls-certs\") pod \"e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt\" (UID: \"23d87d5a-27f0-4374-bd2d-66aa4852939c\") " pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:52:35.650334 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.650309    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqgt5\" (UniqueName: \"kubernetes.io/projected/23d87d5a-27f0-4374-bd2d-66aa4852939c-kube-api-access-hqgt5\") pod \"e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt\" (UID: \"23d87d5a-27f0-4374-bd2d-66aa4852939c\") " pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:52:35.716162 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.716074    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:52:35.986776 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:35.986750    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"]
Sep 11 12:52:35.988836 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:52:35.988808    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23d87d5a_27f0_4374_bd2d_66aa4852939c.slice/crio-0bfc99d1148cae8974cd0c9cedf0e4915d1267f7c64fe2f5d4afbe90851987bf WatchSource:0}: Error finding container 0bfc99d1148cae8974cd0c9cedf0e4915d1267f7c64fe2f5d4afbe90851987bf: Status 404 returned error can't find the container with id 0bfc99d1148cae8974cd0c9cedf0e4915d1267f7c64fe2f5d4afbe90851987bf
Sep 11 12:52:36.723884 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:36.723849    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt" event={"ID":"23d87d5a-27f0-4374-bd2d-66aa4852939c","Type":"ContainerStarted","Data":"74031d24315980e1457c946da69f3fa0b84b3c8d0372e3a2968340bae388127f"}
Sep 11 12:52:36.723884 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:36.723891    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt" event={"ID":"23d87d5a-27f0-4374-bd2d-66aa4852939c","Type":"ContainerStarted","Data":"0bfc99d1148cae8974cd0c9cedf0e4915d1267f7c64fe2f5d4afbe90851987bf"}
Sep 11 12:52:36.743178 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:36.743132    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt" podStartSLOduration=1.743117051 podStartE2EDuration="1.743117051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:52:36.742733712 +0000 UTC m=+2288.305513476" watchObservedRunningTime="2026-09-11 12:52:36.743117051 +0000 UTC m=+2288.305896814"
Sep 11 12:52:37.737405 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:37.737373    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:52:45.716770 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:45.716726    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:52:45.717305 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:45.716785    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:52:45.729420 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:45.729390    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:52:45.774180 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:45.774152    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:52:54.735430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:52:54.735391    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:53:16.713799 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:16.713767    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"]
Sep 11 12:53:18.643281 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:18.643243    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:53:26.125258 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:26.125169    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:53:31.732404 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:31.732339    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt" podUID="23d87d5a-27f0-4374-bd2d-66aa4852939c" containerName="main" containerID="cri-o://74031d24315980e1457c946da69f3fa0b84b3c8d0372e3a2968340bae388127f" gracePeriod=45
Sep 11 12:53:31.972354 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:31.972332    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:53:31.976123 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:31.976095    2561 generic.go:356] "Generic (PLEG): container finished" podID="23d87d5a-27f0-4374-bd2d-66aa4852939c" containerID="74031d24315980e1457c946da69f3fa0b84b3c8d0372e3a2968340bae388127f" exitCode=0
Sep 11 12:53:31.976249 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:31.976135    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"
Sep 11 12:53:31.976249 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:31.976175    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt" event={"ID":"23d87d5a-27f0-4374-bd2d-66aa4852939c","Type":"ContainerDied","Data":"74031d24315980e1457c946da69f3fa0b84b3c8d0372e3a2968340bae388127f"}
Sep 11 12:53:31.976249 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:31.976236    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt" event={"ID":"23d87d5a-27f0-4374-bd2d-66aa4852939c","Type":"ContainerDied","Data":"0bfc99d1148cae8974cd0c9cedf0e4915d1267f7c64fe2f5d4afbe90851987bf"}
Sep 11 12:53:31.976372 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:31.976255    2561 scope.go:117] "RemoveContainer" containerID="74031d24315980e1457c946da69f3fa0b84b3c8d0372e3a2968340bae388127f"
Sep 11 12:53:31.984075 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:31.984066    2561 scope.go:117] "RemoveContainer" containerID="74031d24315980e1457c946da69f3fa0b84b3c8d0372e3a2968340bae388127f"
Sep 11 12:53:31.984440 ip-10-0-131-172 kubenswrapper[2561]: E0911 12:53:31.984419    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74031d24315980e1457c946da69f3fa0b84b3c8d0372e3a2968340bae388127f\": container with ID starting with 74031d24315980e1457c946da69f3fa0b84b3c8d0372e3a2968340bae388127f not found: ID does not exist" containerID="74031d24315980e1457c946da69f3fa0b84b3c8d0372e3a2968340bae388127f"
Sep 11 12:53:31.984511 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:31.984447    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74031d24315980e1457c946da69f3fa0b84b3c8d0372e3a2968340bae388127f"} err="failed to get container status \"74031d24315980e1457c946da69f3fa0b84b3c8d0372e3a2968340bae388127f\": rpc error: code = NotFound desc = could not find container \"74031d24315980e1457c946da69f3fa0b84b3c8d0372e3a2968340bae388127f\": container with ID starting with 74031d24315980e1457c946da69f3fa0b84b3c8d0372e3a2968340bae388127f not found: ID does not exist"
Sep 11 12:53:32.123206 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:32.123156    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23d87d5a-27f0-4374-bd2d-66aa4852939c-tmp-dir\") pod \"23d87d5a-27f0-4374-bd2d-66aa4852939c\" (UID: \"23d87d5a-27f0-4374-bd2d-66aa4852939c\") "
Sep 11 12:53:32.123206 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:32.123206    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23d87d5a-27f0-4374-bd2d-66aa4852939c-dshm\") pod \"23d87d5a-27f0-4374-bd2d-66aa4852939c\" (UID: \"23d87d5a-27f0-4374-bd2d-66aa4852939c\") "
Sep 11 12:53:32.123423 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:32.123258    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqgt5\" (UniqueName: \"kubernetes.io/projected/23d87d5a-27f0-4374-bd2d-66aa4852939c-kube-api-access-hqgt5\") pod \"23d87d5a-27f0-4374-bd2d-66aa4852939c\" (UID: \"23d87d5a-27f0-4374-bd2d-66aa4852939c\") "
Sep 11 12:53:32.123423 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:32.123283    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23d87d5a-27f0-4374-bd2d-66aa4852939c-model-cache\") pod \"23d87d5a-27f0-4374-bd2d-66aa4852939c\" (UID: \"23d87d5a-27f0-4374-bd2d-66aa4852939c\") "
Sep 11 12:53:32.123423 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:32.123301    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23d87d5a-27f0-4374-bd2d-66aa4852939c-tls-certs\") pod \"23d87d5a-27f0-4374-bd2d-66aa4852939c\" (UID: \"23d87d5a-27f0-4374-bd2d-66aa4852939c\") "
Sep 11 12:53:32.123423 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:32.123320    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23d87d5a-27f0-4374-bd2d-66aa4852939c-home\") pod \"23d87d5a-27f0-4374-bd2d-66aa4852939c\" (UID: \"23d87d5a-27f0-4374-bd2d-66aa4852939c\") "
Sep 11 12:53:32.123634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:32.123442    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d87d5a-27f0-4374-bd2d-66aa4852939c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "23d87d5a-27f0-4374-bd2d-66aa4852939c" (UID: "23d87d5a-27f0-4374-bd2d-66aa4852939c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:53:32.123634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:32.123532    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d87d5a-27f0-4374-bd2d-66aa4852939c-model-cache" (OuterVolumeSpecName: "model-cache") pod "23d87d5a-27f0-4374-bd2d-66aa4852939c" (UID: "23d87d5a-27f0-4374-bd2d-66aa4852939c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:53:32.123634 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:32.123624    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d87d5a-27f0-4374-bd2d-66aa4852939c-home" (OuterVolumeSpecName: "home") pod "23d87d5a-27f0-4374-bd2d-66aa4852939c" (UID: "23d87d5a-27f0-4374-bd2d-66aa4852939c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:53:32.125581 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:32.125554    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d87d5a-27f0-4374-bd2d-66aa4852939c-dshm" (OuterVolumeSpecName: "dshm") pod "23d87d5a-27f0-4374-bd2d-66aa4852939c" (UID: "23d87d5a-27f0-4374-bd2d-66aa4852939c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 12:53:32.125581 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:32.125555    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23d87d5a-27f0-4374-bd2d-66aa4852939c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23d87d5a-27f0-4374-bd2d-66aa4852939c" (UID: "23d87d5a-27f0-4374-bd2d-66aa4852939c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:53:32.125738 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:32.125661    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d87d5a-27f0-4374-bd2d-66aa4852939c-kube-api-access-hqgt5" (OuterVolumeSpecName: "kube-api-access-hqgt5") pod "23d87d5a-27f0-4374-bd2d-66aa4852939c" (UID: "23d87d5a-27f0-4374-bd2d-66aa4852939c"). InnerVolumeSpecName "kube-api-access-hqgt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:53:32.224772 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:32.224743    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqgt5\" (UniqueName: \"kubernetes.io/projected/23d87d5a-27f0-4374-bd2d-66aa4852939c-kube-api-access-hqgt5\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:53:32.224772 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:32.224767    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23d87d5a-27f0-4374-bd2d-66aa4852939c-model-cache\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:53:32.224772 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:32.224776    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23d87d5a-27f0-4374-bd2d-66aa4852939c-tls-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:53:32.224978 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:32.224783    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23d87d5a-27f0-4374-bd2d-66aa4852939c-home\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:53:32.224978 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:32.224791    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23d87d5a-27f0-4374-bd2d-66aa4852939c-tmp-dir\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:53:32.224978 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:32.224798    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23d87d5a-27f0-4374-bd2d-66aa4852939c-dshm\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:53:32.297151 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:32.297121    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"]
Sep 11 12:53:32.302000 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:32.301977    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-614772b9-kserve-799d5b796c-jfzwt"]
Sep 11 12:53:32.932736 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:32.932704    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23d87d5a-27f0-4374-bd2d-66aa4852939c" path="/var/lib/kubelet/pods/23d87d5a-27f0-4374-bd2d-66aa4852939c/volumes"
Sep 11 12:53:35.226607 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.226564    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:53:35.418051 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.418015    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-655857d65c-7t9gv"]
Sep 11 12:53:35.418390 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.418377    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23d87d5a-27f0-4374-bd2d-66aa4852939c" containerName="main"
Sep 11 12:53:35.418441 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.418392    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="23d87d5a-27f0-4374-bd2d-66aa4852939c" containerName="main"
Sep 11 12:53:35.418477 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.418452    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="23d87d5a-27f0-4374-bd2d-66aa4852939c" containerName="main"
Sep 11 12:53:35.421398 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.421382    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-655857d65c-7t9gv"
Sep 11 12:53:35.425387 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.425351    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 12:53:35.425387 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.425382    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 12:53:35.425605 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.425410    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l5tx4\""
Sep 11 12:53:35.425605 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.425352    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 12:53:35.425605 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.425475    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 12:53:35.434575 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.434551    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-655857d65c-7t9gv"]
Sep 11 12:53:35.556907 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.556874    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5z7l\" (UniqueName: \"kubernetes.io/projected/449f589d-7ba0-48ac-ada1-f24566e09b91-kube-api-access-r5z7l\") pod \"maas-api-655857d65c-7t9gv\" (UID: \"449f589d-7ba0-48ac-ada1-f24566e09b91\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-7t9gv"
Sep 11 12:53:35.557094 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.556914    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/449f589d-7ba0-48ac-ada1-f24566e09b91-openshift-service-ca\") pod \"maas-api-655857d65c-7t9gv\" (UID: \"449f589d-7ba0-48ac-ada1-f24566e09b91\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-7t9gv"
Sep 11 12:53:35.557094 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.557052    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/449f589d-7ba0-48ac-ada1-f24566e09b91-maas-api-tls\") pod \"maas-api-655857d65c-7t9gv\" (UID: \"449f589d-7ba0-48ac-ada1-f24566e09b91\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-7t9gv"
Sep 11 12:53:35.557094 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.557085    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/449f589d-7ba0-48ac-ada1-f24566e09b91-metrics-certs\") pod \"maas-api-655857d65c-7t9gv\" (UID: \"449f589d-7ba0-48ac-ada1-f24566e09b91\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-7t9gv"
Sep 11 12:53:35.658321 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.658291    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/449f589d-7ba0-48ac-ada1-f24566e09b91-maas-api-tls\") pod \"maas-api-655857d65c-7t9gv\" (UID: \"449f589d-7ba0-48ac-ada1-f24566e09b91\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-7t9gv"
Sep 11 12:53:35.658321 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.658321    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/449f589d-7ba0-48ac-ada1-f24566e09b91-metrics-certs\") pod \"maas-api-655857d65c-7t9gv\" (UID: \"449f589d-7ba0-48ac-ada1-f24566e09b91\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-7t9gv"
Sep 11 12:53:35.658522 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.658341    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5z7l\" (UniqueName: \"kubernetes.io/projected/449f589d-7ba0-48ac-ada1-f24566e09b91-kube-api-access-r5z7l\") pod \"maas-api-655857d65c-7t9gv\" (UID: \"449f589d-7ba0-48ac-ada1-f24566e09b91\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-7t9gv"
Sep 11 12:53:35.658522 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.658362    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/449f589d-7ba0-48ac-ada1-f24566e09b91-openshift-service-ca\") pod \"maas-api-655857d65c-7t9gv\" (UID: \"449f589d-7ba0-48ac-ada1-f24566e09b91\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-7t9gv"
Sep 11 12:53:35.659010 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.658987    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/449f589d-7ba0-48ac-ada1-f24566e09b91-openshift-service-ca\") pod \"maas-api-655857d65c-7t9gv\" (UID: \"449f589d-7ba0-48ac-ada1-f24566e09b91\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-7t9gv"
Sep 11 12:53:35.660853 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.660833    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/449f589d-7ba0-48ac-ada1-f24566e09b91-maas-api-tls\") pod \"maas-api-655857d65c-7t9gv\" (UID: \"449f589d-7ba0-48ac-ada1-f24566e09b91\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-7t9gv"
Sep 11 12:53:35.660937 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.660869    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/449f589d-7ba0-48ac-ada1-f24566e09b91-metrics-certs\") pod \"maas-api-655857d65c-7t9gv\" (UID: \"449f589d-7ba0-48ac-ada1-f24566e09b91\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-7t9gv"
Sep 11 12:53:35.666056 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.666037    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5z7l\" (UniqueName: \"kubernetes.io/projected/449f589d-7ba0-48ac-ada1-f24566e09b91-kube-api-access-r5z7l\") pod \"maas-api-655857d65c-7t9gv\" (UID: \"449f589d-7ba0-48ac-ada1-f24566e09b91\") " pod="odh-ai-gateway-infra/maas-api-655857d65c-7t9gv"
Sep 11 12:53:35.732064 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:35.732036    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-655857d65c-7t9gv"
Sep 11 12:53:36.003000 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:36.002972    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-655857d65c-7t9gv"]
Sep 11 12:53:36.004815 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:53:36.004786    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod449f589d_7ba0_48ac_ada1_f24566e09b91.slice/crio-eda7fc43f0c738a65ac51008a036d9c1f49a70f0294cf9f0b4a27708e7713bf4 WatchSource:0}: Error finding container eda7fc43f0c738a65ac51008a036d9c1f49a70f0294cf9f0b4a27708e7713bf4: Status 404 returned error can't find the container with id eda7fc43f0c738a65ac51008a036d9c1f49a70f0294cf9f0b4a27708e7713bf4
Sep 11 12:53:36.006096 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:36.006079    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 12:53:36.999953 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:36.999920    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-655857d65c-7t9gv" event={"ID":"449f589d-7ba0-48ac-ada1-f24566e09b91","Type":"ContainerStarted","Data":"bc412ae681ea773bc9832f1f8b1537bcd229d4715c284be51f35e0cbe7def177"}
Sep 11 12:53:37.000375 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:36.999966    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-655857d65c-7t9gv" event={"ID":"449f589d-7ba0-48ac-ada1-f24566e09b91","Type":"ContainerStarted","Data":"eda7fc43f0c738a65ac51008a036d9c1f49a70f0294cf9f0b4a27708e7713bf4"}
Sep 11 12:53:37.000375 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:36.999998    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-655857d65c-7t9gv"
Sep 11 12:53:37.018430 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:37.018383    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-655857d65c-7t9gv" podStartSLOduration=1.675622888 podStartE2EDuration="2.018365928s" totalImagesPullingTime="342.74304ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:53:37.017388192 +0000 UTC m=+2348.580167947" watchObservedRunningTime="2026-09-11 12:53:37.018365928 +0000 UTC m=+2348.581145718"
Sep 11 12:53:41.733008 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:41.732970    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:53:42.803450 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:42.803413    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-655857d65c-7t9gv"]
Sep 11 12:53:42.803859 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:42.803605    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-655857d65c-7t9gv" podUID="449f589d-7ba0-48ac-ada1-f24566e09b91" containerName="maas-api" containerID="cri-o://bc412ae681ea773bc9832f1f8b1537bcd229d4715c284be51f35e0cbe7def177" gracePeriod=30
Sep 11 12:53:42.809423 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:42.809399    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-655857d65c-7t9gv"
Sep 11 12:53:42.842772 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:42.842745    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-57b85b5ff7-mk65m"]
Sep 11 12:53:42.846004 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:42.845981    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-mk65m"
Sep 11 12:53:42.860656 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:42.860632    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57b85b5ff7-mk65m"]
Sep 11 12:53:43.018310 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.018275    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce-metrics-certs\") pod \"maas-api-57b85b5ff7-mk65m\" (UID: \"0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce\") " pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-mk65m"
Sep 11 12:53:43.018475 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.018327    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hxlc\" (UniqueName: \"kubernetes.io/projected/0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce-kube-api-access-4hxlc\") pod \"maas-api-57b85b5ff7-mk65m\" (UID: \"0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce\") " pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-mk65m"
Sep 11 12:53:43.018475 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.018394    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce-openshift-service-ca\") pod \"maas-api-57b85b5ff7-mk65m\" (UID: \"0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce\") " pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-mk65m"
Sep 11 12:53:43.018475 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.018461    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce-maas-api-tls\") pod \"maas-api-57b85b5ff7-mk65m\" (UID: \"0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce\") " pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-mk65m"
Sep 11 12:53:43.034168 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.034143    2561 generic.go:356] "Generic (PLEG): container finished" podID="449f589d-7ba0-48ac-ada1-f24566e09b91" containerID="bc412ae681ea773bc9832f1f8b1537bcd229d4715c284be51f35e0cbe7def177" exitCode=0
Sep 11 12:53:43.034293 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.034182    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-655857d65c-7t9gv" event={"ID":"449f589d-7ba0-48ac-ada1-f24566e09b91","Type":"ContainerDied","Data":"bc412ae681ea773bc9832f1f8b1537bcd229d4715c284be51f35e0cbe7def177"}
Sep 11 12:53:43.034293 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.034243    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-655857d65c-7t9gv" event={"ID":"449f589d-7ba0-48ac-ada1-f24566e09b91","Type":"ContainerDied","Data":"eda7fc43f0c738a65ac51008a036d9c1f49a70f0294cf9f0b4a27708e7713bf4"}
Sep 11 12:53:43.034293 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.034259    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eda7fc43f0c738a65ac51008a036d9c1f49a70f0294cf9f0b4a27708e7713bf4"
Sep 11 12:53:43.045118 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.045098    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-655857d65c-7t9gv"
Sep 11 12:53:43.119769 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.119692    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce-metrics-certs\") pod \"maas-api-57b85b5ff7-mk65m\" (UID: \"0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce\") " pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-mk65m"
Sep 11 12:53:43.119769 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.119733    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hxlc\" (UniqueName: \"kubernetes.io/projected/0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce-kube-api-access-4hxlc\") pod \"maas-api-57b85b5ff7-mk65m\" (UID: \"0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce\") " pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-mk65m"
Sep 11 12:53:43.119769 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.119761    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce-openshift-service-ca\") pod \"maas-api-57b85b5ff7-mk65m\" (UID: \"0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce\") " pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-mk65m"
Sep 11 12:53:43.119992 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.119798    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce-maas-api-tls\") pod \"maas-api-57b85b5ff7-mk65m\" (UID: \"0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce\") " pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-mk65m"
Sep 11 12:53:43.120537 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.120514    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce-openshift-service-ca\") pod \"maas-api-57b85b5ff7-mk65m\" (UID: \"0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce\") " pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-mk65m"
Sep 11 12:53:43.122292 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.122268    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce-metrics-certs\") pod \"maas-api-57b85b5ff7-mk65m\" (UID: \"0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce\") " pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-mk65m"
Sep 11 12:53:43.122434 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.122413    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce-maas-api-tls\") pod \"maas-api-57b85b5ff7-mk65m\" (UID: \"0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce\") " pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-mk65m"
Sep 11 12:53:43.128095 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.128072    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hxlc\" (UniqueName: \"kubernetes.io/projected/0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce-kube-api-access-4hxlc\") pod \"maas-api-57b85b5ff7-mk65m\" (UID: \"0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce\") " pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-mk65m"
Sep 11 12:53:43.178624 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.178596    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-mk65m"
Sep 11 12:53:43.220420 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.220390    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/449f589d-7ba0-48ac-ada1-f24566e09b91-metrics-certs\") pod \"449f589d-7ba0-48ac-ada1-f24566e09b91\" (UID: \"449f589d-7ba0-48ac-ada1-f24566e09b91\") "
Sep 11 12:53:43.220583 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.220522    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5z7l\" (UniqueName: \"kubernetes.io/projected/449f589d-7ba0-48ac-ada1-f24566e09b91-kube-api-access-r5z7l\") pod \"449f589d-7ba0-48ac-ada1-f24566e09b91\" (UID: \"449f589d-7ba0-48ac-ada1-f24566e09b91\") "
Sep 11 12:53:43.220583 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.220550    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/449f589d-7ba0-48ac-ada1-f24566e09b91-maas-api-tls\") pod \"449f589d-7ba0-48ac-ada1-f24566e09b91\" (UID: \"449f589d-7ba0-48ac-ada1-f24566e09b91\") "
Sep 11 12:53:43.220694 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.220597    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/449f589d-7ba0-48ac-ada1-f24566e09b91-openshift-service-ca\") pod \"449f589d-7ba0-48ac-ada1-f24566e09b91\" (UID: \"449f589d-7ba0-48ac-ada1-f24566e09b91\") "
Sep 11 12:53:43.221082 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.221049    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/449f589d-7ba0-48ac-ada1-f24566e09b91-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "449f589d-7ba0-48ac-ada1-f24566e09b91" (UID: "449f589d-7ba0-48ac-ada1-f24566e09b91"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 12:53:43.222503 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.222481    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/449f589d-7ba0-48ac-ada1-f24566e09b91-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "449f589d-7ba0-48ac-ada1-f24566e09b91" (UID: "449f589d-7ba0-48ac-ada1-f24566e09b91"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:53:43.222587 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.222512    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449f589d-7ba0-48ac-ada1-f24566e09b91-kube-api-access-r5z7l" (OuterVolumeSpecName: "kube-api-access-r5z7l") pod "449f589d-7ba0-48ac-ada1-f24566e09b91" (UID: "449f589d-7ba0-48ac-ada1-f24566e09b91"). InnerVolumeSpecName "kube-api-access-r5z7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 12:53:43.222587 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.222537    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/449f589d-7ba0-48ac-ada1-f24566e09b91-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "449f589d-7ba0-48ac-ada1-f24566e09b91" (UID: "449f589d-7ba0-48ac-ada1-f24566e09b91"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 12:53:43.322028 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.321996    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/449f589d-7ba0-48ac-ada1-f24566e09b91-metrics-certs\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:53:43.322028 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.322024    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5z7l\" (UniqueName: \"kubernetes.io/projected/449f589d-7ba0-48ac-ada1-f24566e09b91-kube-api-access-r5z7l\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:53:43.322028 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.322035    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/449f589d-7ba0-48ac-ada1-f24566e09b91-maas-api-tls\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:53:43.322283 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.322045    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/449f589d-7ba0-48ac-ada1-f24566e09b91-openshift-service-ca\") on node \"ip-10-0-131-172.ec2.internal\" DevicePath \"\""
Sep 11 12:53:43.449611 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:43.449586    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57b85b5ff7-mk65m"]
Sep 11 12:53:43.451730 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:53:43.451702    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f15dcc8_0359_4c03_9c4b_d85ad2f8b4ce.slice/crio-6b52b4c6bc9f57b05dbaf5440d4ce9b2f4d0c8f9a9179e96cba7a6b3fce8662e WatchSource:0}: Error finding container 6b52b4c6bc9f57b05dbaf5440d4ce9b2f4d0c8f9a9179e96cba7a6b3fce8662e: Status 404 returned error can't find the container with id 6b52b4c6bc9f57b05dbaf5440d4ce9b2f4d0c8f9a9179e96cba7a6b3fce8662e
Sep 11 12:53:44.041129 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:44.041099    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-655857d65c-7t9gv"
Sep 11 12:53:44.041512 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:44.041099    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-mk65m" event={"ID":"0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce","Type":"ContainerStarted","Data":"191306c7b805cefa1ec16f59a73773ae848cde77cbf73614e7f2cfaed9c28696"}
Sep 11 12:53:44.041512 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:44.041233    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-mk65m" event={"ID":"0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce","Type":"ContainerStarted","Data":"6b52b4c6bc9f57b05dbaf5440d4ce9b2f4d0c8f9a9179e96cba7a6b3fce8662e"}
Sep 11 12:53:44.041512 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:44.041396    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-mk65m"
Sep 11 12:53:44.059978 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:44.059936    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-mk65m" podStartSLOduration=1.716779939 podStartE2EDuration="2.059921057s" totalImagesPullingTime="343.141118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:53:44.058599385 +0000 UTC m=+2355.621379152" watchObservedRunningTime="2026-09-11 12:53:44.059921057 +0000 UTC m=+2355.622700838"
Sep 11 12:53:44.074739 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:44.074713    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-655857d65c-7t9gv"]
Sep 11 12:53:44.080085 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:44.080064    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-655857d65c-7t9gv"]
Sep 11 12:53:44.932416 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:44.932380    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="449f589d-7ba0-48ac-ada1-f24566e09b91" path="/var/lib/kubelet/pods/449f589d-7ba0-48ac-ada1-f24566e09b91/volumes"
Sep 11 12:53:47.338176 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:47.338140    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:53:50.051214 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:50.051171    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-57b85b5ff7-mk65m"
Sep 11 12:53:53.047297 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:53.047261    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:53:59.833220 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:53:59.832382    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2g9lc"]
Sep 11 12:54:03.296491 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:03.296454    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9c68dcdc5-dz8c4_1ff9a731-1667-4747-9f10-351e5ac7f0df/authorino/0.log"
Sep 11 12:54:09.290241 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:09.290212    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8dd64745b-xxnpg_db95a0f5-e92b-4bed-b625-c402af9c93d0/copy-manifests/0.log"
Sep 11 12:54:09.297267 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:09.297245    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8dd64745b-xxnpg_db95a0f5-e92b-4bed-b625-c402af9c93d0/manager/0.log"
Sep 11 12:54:09.863096 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:09.863068    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-57b85b5ff7-mk65m_0f15dcc8-0359-4c03-9c4b-d85ad2f8b4ce/maas-api/0.log"
Sep 11 12:54:10.924499 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:10.924469    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9c68dcdc5-dz8c4_1ff9a731-1667-4747-9f10-351e5ac7f0df/authorino/0.log"
Sep 11 12:54:11.259488 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:11.259464    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-pf96w_2a372393-87e4-4660-a47b-5e2a0cefe67d/kuadrant-console-plugin/0.log"
Sep 11 12:54:11.512677 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:11.512601    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2g9lc_9408ef02-4f93-4e61-88c8-39ef23bf0761/limitador/0.log"
Sep 11 12:54:12.205795 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:12.205761    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b5f595c8f-7cjqr_f25fe8ac-8170-4f84-a77c-8f077fe8b458/kube-auth-proxy/0.log"
Sep 11 12:54:13.481130 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:13.481102    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-zrfn7_c9ab1431-d2e2-4cd5-b2b6-900ddf211117/main/0.log"
Sep 11 12:54:13.813281 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:13.813171    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-h6c9z_b270b5fd-d777-4236-9c22-228a4fdfe291/main/0.log"
Sep 11 12:54:21.964561 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:21.964530    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6qbjc_b254d6a2-6adc-468d-b21c-fb5ca9232417/global-pull-secret-syncer/0.log"
Sep 11 12:54:22.157948 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:22.157914    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zbbqf_28c2ca1a-9bbb-4cb5-a226-69eb4db972a2/konnectivity-agent/0.log"
Sep 11 12:54:22.181862 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:22.181840    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-172.ec2.internal_4d587c23d1130ab850ab1128833ea53c/haproxy/0.log"
Sep 11 12:54:27.002861 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:27.002827    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9c68dcdc5-dz8c4_1ff9a731-1667-4747-9f10-351e5ac7f0df/authorino/0.log"
Sep 11 12:54:27.103570 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:27.103545    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-pf96w_2a372393-87e4-4660-a47b-5e2a0cefe67d/kuadrant-console-plugin/0.log"
Sep 11 12:54:27.185675 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:27.185645    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2g9lc_9408ef02-4f93-4e61-88c8-39ef23bf0761/limitador/0.log"
Sep 11 12:54:29.027290 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:29.027233    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-555dfb9dd8-9lkfd_e8ac9128-16ec-4f9c-ac2c-bc3815231d52/metrics-server/0.log"
Sep 11 12:54:29.042510 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:29.042488    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcfm6_88402687-4e5b-411c-be7a-4cb955f90425/ovn-acl-logging/0.log"
Sep 11 12:54:29.049617 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:29.049599    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bcfm6_88402687-4e5b-411c-be7a-4cb955f90425/ovn-acl-logging/0.log"
Sep 11 12:54:29.317421 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:29.317338    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wxcr6_653ad827-71d7-4cc1-bacd-ea44b590787c/node-exporter/0.log"
Sep 11 12:54:29.345085 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:29.345056    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wxcr6_653ad827-71d7-4cc1-bacd-ea44b590787c/kube-rbac-proxy/0.log"
Sep 11 12:54:29.367816 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:29.367795    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wxcr6_653ad827-71d7-4cc1-bacd-ea44b590787c/init-textfile/0.log"
Sep 11 12:54:29.374214 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:29.374179    2561 scope.go:117] "RemoveContainer" containerID="782a1dcaf9972d8639a5afdf37738149b8422c9b54feedffcd8ae5d5f0e0d7c7"
Sep 11 12:54:29.381895 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:29.381878    2561 scope.go:117] "RemoveContainer" containerID="434cc1e3192c4c56ccfdf612252424ca1bf653fe80bc4c18ff1772c579d386ac"
Sep 11 12:54:29.809480 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:29.809449    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-fcd644666-nqc85_ead135a5-e594-468d-b742-c6471543656c/telemeter-client/0.log"
Sep 11 12:54:29.838774 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:29.838744    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-fcd644666-nqc85_ead135a5-e594-468d-b742-c6471543656c/reload/0.log"
Sep 11 12:54:29.865765 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:29.865744    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-fcd644666-nqc85_ead135a5-e594-468d-b742-c6471543656c/kube-rbac-proxy/0.log"
Sep 11 12:54:29.899764 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:29.899742    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59675fdc88-d92dw_32e2be1f-f47c-4158-bdb3-de6e93f42cf9/thanos-query/0.log"
Sep 11 12:54:29.928980 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:29.928952    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59675fdc88-d92dw_32e2be1f-f47c-4158-bdb3-de6e93f42cf9/kube-rbac-proxy-web/0.log"
Sep 11 12:54:29.952427 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:29.952407    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59675fdc88-d92dw_32e2be1f-f47c-4158-bdb3-de6e93f42cf9/kube-rbac-proxy/0.log"
Sep 11 12:54:29.976999 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:29.976981    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59675fdc88-d92dw_32e2be1f-f47c-4158-bdb3-de6e93f42cf9/prom-label-proxy/0.log"
Sep 11 12:54:30.001035 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:30.001017    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59675fdc88-d92dw_32e2be1f-f47c-4158-bdb3-de6e93f42cf9/kube-rbac-proxy-rules/0.log"
Sep 11 12:54:30.034526 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:30.034506    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59675fdc88-d92dw_32e2be1f-f47c-4158-bdb3-de6e93f42cf9/kube-rbac-proxy-metrics/0.log"
Sep 11 12:54:31.390198 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:31.390153    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-n9nrp_188f0ec4-970e-4d5b-85ea-a4cdd0f3b106/networking-console-plugin/0.log"
Sep 11 12:54:32.318372 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.318338    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47"]
Sep 11 12:54:32.318665 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.318654    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="449f589d-7ba0-48ac-ada1-f24566e09b91" containerName="maas-api"
Sep 11 12:54:32.318714 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.318666    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="449f589d-7ba0-48ac-ada1-f24566e09b91" containerName="maas-api"
Sep 11 12:54:32.318750 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.318729    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="449f589d-7ba0-48ac-ada1-f24566e09b91" containerName="maas-api"
Sep 11 12:54:32.323349 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.323333    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47"
Sep 11 12:54:32.326033 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.326012    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zkgjl\"/\"openshift-service-ca.crt\""
Sep 11 12:54:32.326167 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.326033    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zkgjl\"/\"default-dockercfg-dl5jf\""
Sep 11 12:54:32.327161 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.327139    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zkgjl\"/\"kube-root-ca.crt\""
Sep 11 12:54:32.336796 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.336776    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47"]
Sep 11 12:54:32.346371 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.346351    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/25fc6208-b902-4fdd-b55b-64d1d7010297-proc\") pod \"perf-node-gather-daemonset-rpw47\" (UID: \"25fc6208-b902-4fdd-b55b-64d1d7010297\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47"
Sep 11 12:54:32.346474 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.346388    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25fc6208-b902-4fdd-b55b-64d1d7010297-sys\") pod \"perf-node-gather-daemonset-rpw47\" (UID: \"25fc6208-b902-4fdd-b55b-64d1d7010297\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47"
Sep 11 12:54:32.346474 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.346417    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/25fc6208-b902-4fdd-b55b-64d1d7010297-podres\") pod \"perf-node-gather-daemonset-rpw47\" (UID: \"25fc6208-b902-4fdd-b55b-64d1d7010297\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47"
Sep 11 12:54:32.346552 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.346484    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6rhj\" (UniqueName: \"kubernetes.io/projected/25fc6208-b902-4fdd-b55b-64d1d7010297-kube-api-access-p6rhj\") pod \"perf-node-gather-daemonset-rpw47\" (UID: \"25fc6208-b902-4fdd-b55b-64d1d7010297\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47"
Sep 11 12:54:32.346552 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.346507    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25fc6208-b902-4fdd-b55b-64d1d7010297-lib-modules\") pod \"perf-node-gather-daemonset-rpw47\" (UID: \"25fc6208-b902-4fdd-b55b-64d1d7010297\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47"
Sep 11 12:54:32.447303 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.447260    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6rhj\" (UniqueName: \"kubernetes.io/projected/25fc6208-b902-4fdd-b55b-64d1d7010297-kube-api-access-p6rhj\") pod \"perf-node-gather-daemonset-rpw47\" (UID: \"25fc6208-b902-4fdd-b55b-64d1d7010297\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47"
Sep 11 12:54:32.447303 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.447310    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25fc6208-b902-4fdd-b55b-64d1d7010297-lib-modules\") pod \"perf-node-gather-daemonset-rpw47\" (UID: \"25fc6208-b902-4fdd-b55b-64d1d7010297\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47"
Sep 11 12:54:32.447756 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.447343    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/25fc6208-b902-4fdd-b55b-64d1d7010297-proc\") pod \"perf-node-gather-daemonset-rpw47\" (UID: \"25fc6208-b902-4fdd-b55b-64d1d7010297\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47"
Sep 11 12:54:32.447756 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.447383    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25fc6208-b902-4fdd-b55b-64d1d7010297-sys\") pod \"perf-node-gather-daemonset-rpw47\" (UID: \"25fc6208-b902-4fdd-b55b-64d1d7010297\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47"
Sep 11 12:54:32.447756 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.447421    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/25fc6208-b902-4fdd-b55b-64d1d7010297-podres\") pod \"perf-node-gather-daemonset-rpw47\" (UID: \"25fc6208-b902-4fdd-b55b-64d1d7010297\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47"
Sep 11 12:54:32.447756 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.447478    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/25fc6208-b902-4fdd-b55b-64d1d7010297-proc\") pod \"perf-node-gather-daemonset-rpw47\" (UID: \"25fc6208-b902-4fdd-b55b-64d1d7010297\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47"
Sep 11 12:54:32.447756 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.447500    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25fc6208-b902-4fdd-b55b-64d1d7010297-sys\") pod \"perf-node-gather-daemonset-rpw47\" (UID: \"25fc6208-b902-4fdd-b55b-64d1d7010297\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47"
Sep 11 12:54:32.447756 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.447510    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25fc6208-b902-4fdd-b55b-64d1d7010297-lib-modules\") pod \"perf-node-gather-daemonset-rpw47\" (UID: \"25fc6208-b902-4fdd-b55b-64d1d7010297\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47"
Sep 11 12:54:32.447756 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.447537    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/25fc6208-b902-4fdd-b55b-64d1d7010297-podres\") pod \"perf-node-gather-daemonset-rpw47\" (UID: \"25fc6208-b902-4fdd-b55b-64d1d7010297\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47"
Sep 11 12:54:32.458963 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.458927    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6rhj\" (UniqueName: \"kubernetes.io/projected/25fc6208-b902-4fdd-b55b-64d1d7010297-kube-api-access-p6rhj\") pod \"perf-node-gather-daemonset-rpw47\" (UID: \"25fc6208-b902-4fdd-b55b-64d1d7010297\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47"
Sep 11 12:54:32.633622 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.633523    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47"
Sep 11 12:54:32.634898 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.634874    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-67fcb57878-lbbjf_fbf39cdb-14b2-49a7-a2c1-145223f7588a/console/0.log"
Sep 11 12:54:32.911713 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:32.911626    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47"]
Sep 11 12:54:32.915736 ip-10-0-131-172 kubenswrapper[2561]: W0911 12:54:32.915706    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod25fc6208_b902_4fdd_b55b_64d1d7010297.slice/crio-aab25c52bc3d6572526030f576478c932047635f5d8ceda42df57b6f74860093 WatchSource:0}: Error finding container aab25c52bc3d6572526030f576478c932047635f5d8ceda42df57b6f74860093: Status 404 returned error can't find the container with id aab25c52bc3d6572526030f576478c932047635f5d8ceda42df57b6f74860093
Sep 11 12:54:33.268555 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:33.268520    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47" event={"ID":"25fc6208-b902-4fdd-b55b-64d1d7010297","Type":"ContainerStarted","Data":"e21ad2aa066a783782a28590a6eacffb6961425825deaaad19fc9179fa13d4c6"}
Sep 11 12:54:33.268746 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:33.268573    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47" event={"ID":"25fc6208-b902-4fdd-b55b-64d1d7010297","Type":"ContainerStarted","Data":"aab25c52bc3d6572526030f576478c932047635f5d8ceda42df57b6f74860093"}
Sep 11 12:54:33.268746 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:33.268643    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47"
Sep 11 12:54:33.292121 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:33.292081    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47" podStartSLOduration=1.292067241 podStartE2EDuration="1.292067241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:54:33.290922286 +0000 UTC m=+2404.853702046" watchObservedRunningTime="2026-09-11 12:54:33.292067241 +0000 UTC m=+2404.854847004"
Sep 11 12:54:34.563751 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:34.563722    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ptkj7_4e3ebd74-ae34-4842-8ba8-184dc4cb54cb/dns/0.log"
Sep 11 12:54:34.587311 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:34.587282    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ptkj7_4e3ebd74-ae34-4842-8ba8-184dc4cb54cb/kube-rbac-proxy/0.log"
Sep 11 12:54:34.684388 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:34.684359    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cj75q_b5df1e85-4605-4418-a2cb-b88a26b5c5b3/dns-node-resolver/0.log"
Sep 11 12:54:35.351478 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:35.351441    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6986b74764-wcmn5_21a8830b-a81f-488a-bb3e-8ad43abc7d50/registry/0.log"
Sep 11 12:54:35.381782 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:35.381758    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5bxb8_455e42df-85a6-4c25-a449-e12375d132e6/node-ca/0.log"
Sep 11 12:54:36.640716 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:36.640690    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b5f595c8f-7cjqr_f25fe8ac-8170-4f84-a77c-8f077fe8b458/kube-auth-proxy/0.log"
Sep 11 12:54:37.717473 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:37.717437    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n5nml_25006c65-374d-49ed-b247-a8e0e9833b34/serve-healthcheck-canary/0.log"
Sep 11 12:54:38.470736 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:38.470704    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zhfbt_f4ef5ce8-4e5c-4e66-a862-af9d4159263f/kube-rbac-proxy/0.log"
Sep 11 12:54:38.492965 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:38.492941    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zhfbt_f4ef5ce8-4e5c-4e66-a862-af9d4159263f/exporter/0.log"
Sep 11 12:54:38.516263 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:38.516241    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zhfbt_f4ef5ce8-4e5c-4e66-a862-af9d4159263f/extractor/0.log"
Sep 11 12:54:39.282672 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:39.282641    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-rpw47"
Sep 11 12:54:41.514636 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:41.514606    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8dd64745b-xxnpg_db95a0f5-e92b-4bed-b625-c402af9c93d0/manager/0.log"
Sep 11 12:54:41.552758 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:41.552695    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8dd64745b-xxnpg_db95a0f5-e92b-4bed-b625-c402af9c93d0/copy-manifests/0.log"
Sep 11 12:54:42.838744 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:42.838710    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b5f595c8f-7cjqr_f25fe8ac-8170-4f84-a77c-8f077fe8b458/kube-auth-proxy/0.log"
Sep 11 12:54:42.975213 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:42.975176    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-69c967fbf8-sh9sv_a253c393-34c4-4d8a-8179-8474b2af390c/manager/0.log"
Sep 11 12:54:44.757138 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:44.757110    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b5f595c8f-7cjqr_f25fe8ac-8170-4f84-a77c-8f077fe8b458/kube-auth-proxy/0.log"
Sep 11 12:54:48.462698 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:48.462618    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-8mz42_8d390870-ad6f-41e8-b467-f09aecf5adb7/migrator/0.log"
Sep 11 12:54:48.481705 ip-10-0-131-172 kubenswrapper[2561]: I0911 12:54:48.481680    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-8mz42_8d390870-ad6f-41e8-b467-f09aecf5adb7/graceful-termination/0.log"