Sep 08 16:30:15.724107 ip-10-0-139-51 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 16:30:16.113567 ip-10-0-139-51 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 16:30:16.113567 ip-10-0-139-51 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 16:30:16.113567 ip-10-0-139-51 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 16:30:16.113567 ip-10-0-139-51 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 16:30:16.113567 ip-10-0-139-51 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 16:30:16.114262 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.113636    2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 16:30:16.116450 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116434    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 16:30:16.116450 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116449    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 16:30:16.116521 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116453    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 16:30:16.116521 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116456    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 16:30:16.116521 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116459    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 16:30:16.116521 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116462    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 16:30:16.116521 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116465    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 16:30:16.116521 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116467    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 16:30:16.116521 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116470    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 16:30:16.116521 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116473    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 16:30:16.116521 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116475    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 16:30:16.116521 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116479    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 16:30:16.116521 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116482    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 16:30:16.116521 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116484    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 16:30:16.116521 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116487    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 16:30:16.116521 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116489    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 16:30:16.116521 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116492    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 16:30:16.116521 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116495    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 16:30:16.116521 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116497    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 16:30:16.116521 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116500    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 16:30:16.116521 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116503    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 16:30:16.116521 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116505    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 16:30:16.117009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116508    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 16:30:16.117009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116510    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 16:30:16.117009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116513    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 16:30:16.117009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116515    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 16:30:16.117009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116518    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 16:30:16.117009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116520    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 16:30:16.117009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116526    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 16:30:16.117009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116529    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 16:30:16.117009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116532    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 16:30:16.117009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116535    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 16:30:16.117009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116538    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 16:30:16.117009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116541    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 16:30:16.117009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116543    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 16:30:16.117009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116546    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 16:30:16.117009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116549    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 16:30:16.117009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116551    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 16:30:16.117009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116554    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 16:30:16.117009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116558    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 16:30:16.117009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116562    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 16:30:16.117009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116565    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 16:30:16.117492 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116568    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 16:30:16.117492 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116571    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 16:30:16.117492 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116573    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 16:30:16.117492 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116576    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 16:30:16.117492 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116579    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 16:30:16.117492 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116582    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 16:30:16.117492 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116585    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 16:30:16.117492 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116587    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 16:30:16.117492 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116590    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 16:30:16.117492 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116593    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 16:30:16.117492 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116595    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 16:30:16.117492 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116598    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 16:30:16.117492 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116600    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 16:30:16.117492 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116605    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 16:30:16.117492 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116607    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 16:30:16.117492 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116610    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 16:30:16.117492 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116614    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 16:30:16.117492 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116618    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 16:30:16.117492 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116621    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 16:30:16.117985 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116624    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 16:30:16.117985 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116626    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 16:30:16.117985 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116629    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 16:30:16.117985 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116632    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 16:30:16.117985 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116634    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 16:30:16.117985 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116637    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 16:30:16.117985 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116639    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 16:30:16.117985 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116642    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 16:30:16.117985 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116645    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 16:30:16.117985 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116648    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 16:30:16.117985 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116650    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 16:30:16.117985 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116653    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 16:30:16.117985 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116655    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 16:30:16.117985 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116658    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 16:30:16.117985 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116661    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 16:30:16.117985 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116663    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 16:30:16.117985 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116666    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 16:30:16.117985 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116668    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 16:30:16.117985 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116671    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 16:30:16.117985 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116674    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 16:30:16.118464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116676    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 16:30:16.118464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116679    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 16:30:16.118464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116681    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 16:30:16.118464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116684    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 16:30:16.118464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116686    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 16:30:16.118464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116690    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 16:30:16.118464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116694    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 16:30:16.118464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116698    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 16:30:16.118464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116701    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 16:30:16.118464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116704    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 16:30:16.118464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116706    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 16:30:16.118464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116710    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 16:30:16.118464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116712    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 16:30:16.118464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116715    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 16:30:16.118464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116717    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 16:30:16.118464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116720    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 16:30:16.118464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116723    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 16:30:16.118464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.116725    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 16:30:16.118464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117418    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 16:30:16.118972 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117424    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 16:30:16.118972 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117427    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 16:30:16.118972 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117431    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 16:30:16.118972 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117434    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 16:30:16.118972 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117438    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 16:30:16.118972 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117442    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 16:30:16.118972 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117445    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 16:30:16.118972 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117448    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 16:30:16.118972 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117450    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 16:30:16.118972 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117453    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 16:30:16.118972 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117456    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 16:30:16.118972 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117459    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 16:30:16.118972 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117462    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 16:30:16.118972 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117466    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 16:30:16.118972 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117469    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 16:30:16.118972 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117472    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 16:30:16.118972 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117474    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 16:30:16.118972 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117477    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 16:30:16.118972 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117481    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 16:30:16.118972 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117484    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 16:30:16.119464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117487    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 16:30:16.119464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117490    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 16:30:16.119464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117492    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 16:30:16.119464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117496    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 16:30:16.119464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117500    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 16:30:16.119464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117503    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 16:30:16.119464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117506    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 16:30:16.119464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117508    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 16:30:16.119464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117511    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 16:30:16.119464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117514    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 16:30:16.119464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117517    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 16:30:16.119464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117520    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 16:30:16.119464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117522    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 16:30:16.119464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117525    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 16:30:16.119464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117527    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 16:30:16.119464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117530    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 16:30:16.119464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117532    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 16:30:16.119464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117534    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 16:30:16.119464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117537    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 16:30:16.119464 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117539    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 16:30:16.119991 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117542    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 16:30:16.119991 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117544    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 16:30:16.119991 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117547    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 16:30:16.119991 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117549    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 16:30:16.119991 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117552    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 16:30:16.119991 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117554    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 16:30:16.119991 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117557    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 16:30:16.119991 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117560    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 16:30:16.119991 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117562    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 16:30:16.119991 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117564    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 16:30:16.119991 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117567    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 16:30:16.119991 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117570    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 16:30:16.119991 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117573    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 16:30:16.119991 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117576    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 16:30:16.119991 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117578    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 16:30:16.119991 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117581    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 16:30:16.119991 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117584    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 16:30:16.119991 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117586    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 16:30:16.119991 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117589    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 16:30:16.119991 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117591    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 16:30:16.120485 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117594    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 16:30:16.120485 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117597    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 16:30:16.120485 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117600    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 16:30:16.120485 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117602    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 16:30:16.120485 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117605    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 16:30:16.120485 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117607    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 16:30:16.120485 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117610    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 16:30:16.120485 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117612    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 16:30:16.120485 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117615    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 16:30:16.120485 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117617    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 16:30:16.120485 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117620    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 16:30:16.120485 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117622    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 16:30:16.120485 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117625    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 16:30:16.120485 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117627    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 16:30:16.120485 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117630    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 16:30:16.120485 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117632    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 16:30:16.120485 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117635    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 16:30:16.120485 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117637    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 16:30:16.120485 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117639    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 16:30:16.120970 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117643    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 16:30:16.120970 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117645    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 16:30:16.120970 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117648    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 16:30:16.120970 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117650    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 16:30:16.120970 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117654    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 16:30:16.120970 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117656    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 16:30:16.120970 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117659    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 16:30:16.120970 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117661    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 16:30:16.120970 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117664    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 16:30:16.120970 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117666    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 16:30:16.120970 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117670    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 16:30:16.120970 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117673    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 16:30:16.120970 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117676    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 16:30:16.120970 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117679    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 16:30:16.120970 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117681    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 16:30:16.120970 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117684    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 16:30:16.120970 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117687    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 16:30:16.120970 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117690    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 16:30:16.120970 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.117692    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 16:30:16.121422 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118094    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 16:30:16.121422 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118099    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 16:30:16.121422 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118102    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 16:30:16.121422 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118105    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 16:30:16.121422 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118108    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 16:30:16.121422 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118111    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 16:30:16.121422 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118113    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 16:30:16.121422 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118116    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 16:30:16.121422 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118118    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 16:30:16.121422 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118123    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 16:30:16.121422 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118127    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 16:30:16.121422 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118130    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 16:30:16.121422 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118133    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 16:30:16.121422 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118136    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 16:30:16.121422 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118138    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 16:30:16.121422 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118141    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 16:30:16.121422 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118144    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 16:30:16.121422 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118147    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 16:30:16.121422 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118150    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 16:30:16.122002 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118153    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 16:30:16.122002 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118155    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 16:30:16.122002 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118158    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 16:30:16.122002 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118160    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 16:30:16.122002 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118163    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 16:30:16.122002 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118165    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 16:30:16.122002 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118169    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 16:30:16.122002 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118172    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 16:30:16.122002 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118174    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 16:30:16.122002 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118177    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 16:30:16.122002 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118179    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 16:30:16.122002 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118182    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 16:30:16.122002 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118185    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 16:30:16.122002 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118187    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 16:30:16.122002 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118190    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 16:30:16.122002 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118193    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 16:30:16.122002 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118195    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 16:30:16.122002 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118197    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 16:30:16.122002 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118200    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 16:30:16.122002 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118202    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 16:30:16.122487 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118204    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 16:30:16.122487 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118207    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 16:30:16.122487 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118209    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 16:30:16.122487 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118212    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 16:30:16.122487 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118214    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 16:30:16.122487 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118217    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 16:30:16.122487 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118219    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 16:30:16.122487 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118224    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 16:30:16.122487 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118226    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 16:30:16.122487 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118229    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 16:30:16.122487 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118234    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 16:30:16.122487 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118237    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 16:30:16.122487 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118239    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 16:30:16.122487 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118241    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 16:30:16.122487 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118245    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 16:30:16.122487 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118248    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 16:30:16.122487 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118251    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 16:30:16.122487 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118254    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 16:30:16.122487 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118256    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 16:30:16.122966 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118259    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 16:30:16.122966 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118261    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 16:30:16.122966 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118264    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 16:30:16.122966 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118266    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 16:30:16.122966 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118269    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 16:30:16.122966 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118271    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 16:30:16.122966 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118273    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 16:30:16.122966 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118276    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 16:30:16.122966 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118278    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 16:30:16.122966 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118281    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 16:30:16.122966 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118283    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 16:30:16.122966 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118285    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 16:30:16.122966 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118288    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 16:30:16.122966 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118290    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 16:30:16.122966 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118292    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 16:30:16.122966 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118295    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 16:30:16.122966 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118297    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 16:30:16.122966 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118299    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 16:30:16.122966 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118302    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 16:30:16.122966 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118304    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 16:30:16.123460 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118307    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 16:30:16.123460 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118310    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 16:30:16.123460 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118313    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 16:30:16.123460 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118317    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 16:30:16.123460 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118319    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 16:30:16.123460 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118322    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 16:30:16.123460 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118324    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 16:30:16.123460 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118327    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 16:30:16.123460 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118329    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 16:30:16.123460 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118333    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 16:30:16.123460 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118335    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 16:30:16.123460 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118338    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 16:30:16.123460 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118340    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 16:30:16.123460 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118342    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 16:30:16.123460 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118345    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 16:30:16.123460 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118347    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 16:30:16.123460 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118350    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 16:30:16.123460 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118352    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 16:30:16.123460 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118355    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 16:30:16.123460 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118357    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 16:30:16.123996 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.118360    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 16:30:16.123996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118429    2557 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 16:30:16.123996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118472    2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 16:30:16.123996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118481    2557 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 16:30:16.123996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118485    2557 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 16:30:16.123996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118490    2557 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 16:30:16.123996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118493    2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 16:30:16.123996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118497    2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 16:30:16.123996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118502    2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 16:30:16.123996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118505    2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 16:30:16.123996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118508    2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 16:30:16.123996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118511    2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 16:30:16.123996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118516    2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 16:30:16.123996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118519    2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 16:30:16.123996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118523    2557 flags.go:64] FLAG: --cgroup-root=""
Sep 08 16:30:16.123996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118527    2557 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 16:30:16.123996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118530    2557 flags.go:64] FLAG: --client-ca-file=""
Sep 08 16:30:16.123996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118533    2557 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 16:30:16.123996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118536    2557 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 16:30:16.123996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118540    2557 flags.go:64] FLAG: --cluster-domain=""
Sep 08 16:30:16.123996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118542    2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 16:30:16.123996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118545    2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 16:30:16.123996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118548    2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 16:30:16.123996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118552    2557 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118555    2557 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118558    2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118561    2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118564    2557 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118567    2557 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118570    2557 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118573    2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118576    2557 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118579    2557 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118583    2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118586    2557 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118588    2557 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118591    2557 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118595    2557 flags.go:64] FLAG: --enable-server="true"
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118597    2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118602    2557 flags.go:64] FLAG: --event-burst="100"
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118605    2557 flags.go:64] FLAG: --event-qps="50"
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118608    2557 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118611    2557 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118614    2557 flags.go:64] FLAG: --eviction-hard=""
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118618    2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118621    2557 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118624    2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118628    2557 flags.go:64] FLAG: --eviction-soft=""
Sep 08 16:30:16.124586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118632    2557 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 16:30:16.125231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118634    2557 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 16:30:16.125231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118637    2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 16:30:16.125231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118640    2557 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 16:30:16.125231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118643    2557 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 16:30:16.125231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118646    2557 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 16:30:16.125231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118648    2557 flags.go:64] FLAG: --feature-gates=""
Sep 08 16:30:16.125231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118652    2557 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 16:30:16.125231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118655    2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 16:30:16.125231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118658    2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 16:30:16.125231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118661    2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 16:30:16.125231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118664    2557 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 16:30:16.125231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118666    2557 flags.go:64] FLAG: --help="false"
Sep 08 16:30:16.125231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118669    2557 flags.go:64] FLAG: --hostname-override="ip-10-0-139-51.ec2.internal"
Sep 08 16:30:16.125231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118672    2557 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 16:30:16.125231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118675    2557 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 16:30:16.125231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118678    2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 16:30:16.125231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118681    2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 16:30:16.125231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118684    2557 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 16:30:16.125231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118686    2557 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 16:30:16.125231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118689    2557 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 16:30:16.125231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118692    2557 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 16:30:16.125231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118694    2557 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 16:30:16.125231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118697    2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118700    2557 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118703    2557 flags.go:64] FLAG: --kube-reserved=""
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118706    2557 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118709    2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118714    2557 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118716    2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118719    2557 flags.go:64] FLAG: --lock-file=""
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118723    2557 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118727    2557 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118730    2557 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118735    2557 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118738    2557 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118741    2557 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118744    2557 flags.go:64] FLAG: --logging-format="text"
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118746    2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118750    2557 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118752    2557 flags.go:64] FLAG: --manifest-url=""
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118755    2557 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118760    2557 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118762    2557 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118766    2557 flags.go:64] FLAG: --max-pods="110"
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118769    2557 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118772    2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118775    2557 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 16:30:16.125821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118778    2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118781    2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118783    2557 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118786    2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118794    2557 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118797    2557 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118799    2557 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118802    2557 flags.go:64] FLAG: --pod-cidr=""
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118805    2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118811    2557 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118814    2557 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118817    2557 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118821    2557 flags.go:64] FLAG: --port="10250"
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118824    2557 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118827    2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a8188734a3fb6085"
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118844    2557 flags.go:64] FLAG: --qos-reserved=""
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118848    2557 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118852    2557 flags.go:64] FLAG: --register-node="true"
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118855    2557 flags.go:64] FLAG: --register-with-taints=""
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118859    2557 flags.go:64] FLAG: --registry-burst="10"
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118862    2557 flags.go:64] FLAG: --registry-qps="5"
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118865    2557 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118868    2557 flags.go:64] FLAG: --reserved-memory=""
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118871    2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118874    2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 16:30:16.126446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118877    2557 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118880    2557 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118883    2557 flags.go:64] FLAG: --runonce="false"
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118885    2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118888    2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118891    2557 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118894    2557 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118897    2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118900    2557 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118903    2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118907    2557 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118909    2557 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118912    2557 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118915    2557 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118918    2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118921    2557 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118923    2557 flags.go:64] FLAG: --system-cgroups=""
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118926    2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118931    2557 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118935    2557 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118938    2557 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118941    2557 flags.go:64] FLAG: --tls-min-version=""
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118944    2557 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118947    2557 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118950    2557 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 16:30:16.127082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118954    2557 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 16:30:16.127721 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118956    2557 flags.go:64] FLAG: --v="2"
Sep 08 16:30:16.127721 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118965    2557 flags.go:64] FLAG: --version="false"
Sep 08 16:30:16.127721 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118969    2557 flags.go:64] FLAG: --vmodule=""
Sep 08 16:30:16.127721 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118973    2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 16:30:16.127721 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.118976    2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 16:30:16.127721 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119115    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 16:30:16.127721 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119119    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 16:30:16.127721 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119122    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 16:30:16.127721 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119125    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 16:30:16.127721 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119128    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 16:30:16.127721 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119130    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 16:30:16.127721 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119133    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 16:30:16.127721 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119135    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 16:30:16.127721 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119138    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 16:30:16.127721 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119140    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 16:30:16.127721 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119143    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 16:30:16.127721 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119145    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 16:30:16.127721 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119148    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 16:30:16.127721 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119150    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 16:30:16.127721 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119153    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 16:30:16.127721 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119155    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 16:30:16.128261 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119157    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 16:30:16.128261 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119160    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 16:30:16.128261 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119162    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 16:30:16.128261 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119165    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 16:30:16.128261 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119169    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 16:30:16.128261 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119171    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 16:30:16.128261 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119174    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 16:30:16.128261 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119176    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 16:30:16.128261 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119180    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 16:30:16.128261 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119182    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 16:30:16.128261 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119185    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 16:30:16.128261 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119188    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 16:30:16.128261 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119190    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 16:30:16.128261 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119192    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 16:30:16.128261 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119195    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 16:30:16.128261 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119207    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 16:30:16.128261 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119210    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 16:30:16.128261 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119212    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 16:30:16.128261 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119216    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 16:30:16.128735 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119220    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 16:30:16.128735 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119223    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 16:30:16.128735 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119227    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 16:30:16.128735 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119232    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 16:30:16.128735 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119236    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 16:30:16.128735 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119240    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 16:30:16.128735 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119244    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 16:30:16.128735 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119248    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 16:30:16.128735 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119252    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 16:30:16.128735 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119255    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 16:30:16.128735 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119257    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 16:30:16.128735 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119260    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 16:30:16.128735 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119262    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 16:30:16.128735 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119265    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 16:30:16.128735 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119267    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 16:30:16.128735 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119270    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 16:30:16.128735 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119273    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 16:30:16.128735 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119276    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 16:30:16.128735 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119278    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 16:30:16.128735 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119281    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 16:30:16.129231 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119284    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 16:30:16.129231 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119287    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 16:30:16.129231 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119290    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 16:30:16.129231 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119294    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 16:30:16.129231 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119297    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 16:30:16.129231 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119300    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 16:30:16.129231 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119303    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 16:30:16.129231 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119306    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 16:30:16.129231 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119308    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 16:30:16.129231 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119311    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 16:30:16.129231 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119314    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 16:30:16.129231 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119316    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 16:30:16.129231 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119319    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 16:30:16.129231 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119321    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 16:30:16.129231 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119324    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 16:30:16.129231 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119326    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 16:30:16.129231 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119329    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 16:30:16.129231 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119331    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 16:30:16.129231 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119334    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 16:30:16.129231 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119336    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 16:30:16.129726 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119339    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 16:30:16.129726 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119342    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 16:30:16.129726 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119344    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 16:30:16.129726 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119347    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 16:30:16.129726 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119349    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 16:30:16.129726 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119351    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 16:30:16.129726 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119354    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 16:30:16.129726 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119356    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 16:30:16.129726 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119358    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 16:30:16.129726 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119362    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 16:30:16.129726 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119366    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 16:30:16.129726 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119369    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 16:30:16.129726 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119372    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 16:30:16.129726 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119374    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 16:30:16.129726 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119377    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 16:30:16.129726 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119379    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 16:30:16.129726 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119382    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 16:30:16.129726 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119385    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 16:30:16.129726 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119388    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 16:30:16.130216 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119390    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 16:30:16.130216 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119392    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 16:30:16.130216 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119395    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 16:30:16.130216 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119397    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 16:30:16.130216 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.119400    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 16:30:16.130216 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.119405    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 16:30:16.130216 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.126271    2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 16:30:16.130216 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.126286    2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 16:30:16.130216 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.128821    2557 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 16:30:16.130216 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.128841    2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 16:30:16.130216 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128902    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 16:30:16.130216 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128907    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 16:30:16.130216 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128910    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 16:30:16.130216 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128913    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 16:30:16.130216 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128916    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 16:30:16.130593 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128918    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 16:30:16.130593 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128921    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 16:30:16.130593 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128924    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 16:30:16.130593 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128927    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 16:30:16.130593 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128930    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 16:30:16.130593 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128932    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 16:30:16.130593 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128935    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 16:30:16.130593 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128938    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 16:30:16.130593 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128941    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 16:30:16.130593 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128943    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 16:30:16.130593 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128946    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 16:30:16.130593 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128948    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 16:30:16.130593 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128951    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 16:30:16.130593 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128953    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 16:30:16.130593 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128956    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 16:30:16.130593 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128958    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 16:30:16.130593 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128960    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 16:30:16.130593 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128963    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 16:30:16.130593 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128965    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 16:30:16.130593 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128968    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 16:30:16.131126 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128970    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 16:30:16.131126 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128973    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 16:30:16.131126 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128975    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 16:30:16.131126 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128978    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 16:30:16.131126 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128981    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 16:30:16.131126 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128986    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 16:30:16.131126 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128990    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 16:30:16.131126 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128993    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 16:30:16.131126 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128995    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 16:30:16.131126 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.128998    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 16:30:16.131126 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129001    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 16:30:16.131126 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129004    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 16:30:16.131126 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129007    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 16:30:16.131126 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129009    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 16:30:16.131126 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129012    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 16:30:16.131126 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129015    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 16:30:16.131126 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129017    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 16:30:16.131126 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129020    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 16:30:16.131126 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129023    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 16:30:16.131588 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129025    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 16:30:16.131588 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129028    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 16:30:16.131588 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129031    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 16:30:16.131588 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129033    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 16:30:16.131588 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129036    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 16:30:16.131588 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129039    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 16:30:16.131588 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129042    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 16:30:16.131588 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129044    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 16:30:16.131588 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129047    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 16:30:16.131588 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129050    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 16:30:16.131588 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129052    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 16:30:16.131588 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129054    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 16:30:16.131588 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129057    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 16:30:16.131588 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129060    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 16:30:16.131588 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129062    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 16:30:16.131588 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129065    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 16:30:16.131588 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129067    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 16:30:16.131588 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129070    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 16:30:16.131588 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129073    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 16:30:16.131588 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129075    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 16:30:16.132067 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129078    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 16:30:16.132067 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129081    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 16:30:16.132067 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129084    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 16:30:16.132067 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129088    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 16:30:16.132067 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129091    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 16:30:16.132067 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129094    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 16:30:16.132067 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129096    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 16:30:16.132067 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129100    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 16:30:16.132067 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129104    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 16:30:16.132067 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129106    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 16:30:16.132067 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129109    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 16:30:16.132067 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129112    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 16:30:16.132067 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129115    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 16:30:16.132067 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129118    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 16:30:16.132067 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129121    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 16:30:16.132067 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129124    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 16:30:16.132067 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129126    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 16:30:16.132067 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129128    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 16:30:16.132067 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129131    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 16:30:16.132580 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129133    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 16:30:16.132580 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129136    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 16:30:16.132580 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129138    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 16:30:16.132580 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129140    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 16:30:16.132580 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129143    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 16:30:16.132580 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129146    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 16:30:16.132580 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129148    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 16:30:16.132580 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129151    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 16:30:16.132580 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129154    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 16:30:16.132580 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129156    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 16:30:16.132580 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129159    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 16:30:16.132580 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129161    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 16:30:16.132580 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129164    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 16:30:16.132580 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129166    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 16:30:16.132580 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129169    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 16:30:16.132580 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129172    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 16:30:16.133009 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.129177    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 16:30:16.133009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129289    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 16:30:16.133009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129294    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 16:30:16.133009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129297    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 16:30:16.133009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129299    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 16:30:16.133009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129302    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 16:30:16.133009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129305    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 16:30:16.133009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129309    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 16:30:16.133009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129313    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 16:30:16.133009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129316    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 16:30:16.133009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129318    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 16:30:16.133009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129321    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 16:30:16.133009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129324    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 16:30:16.133009 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129327    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 16:30:16.133359 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129329    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 16:30:16.133359 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129332    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 16:30:16.133359 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129334    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 16:30:16.133359 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129337    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 16:30:16.133359 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129339    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 16:30:16.133359 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129342    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 16:30:16.133359 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129344    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 16:30:16.133359 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129346    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 16:30:16.133359 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129349    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 16:30:16.133359 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129352    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 16:30:16.133359 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129354    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 16:30:16.133359 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129356    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 16:30:16.133359 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129359    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 16:30:16.133359 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129361    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 16:30:16.133359 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129364    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 16:30:16.133359 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129367    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 16:30:16.133359 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129369    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 16:30:16.133359 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129372    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 16:30:16.133359 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129374    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 16:30:16.133359 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129377    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 16:30:16.133845 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129379    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 16:30:16.133845 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129381    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 16:30:16.133845 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129384    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 16:30:16.133845 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129388    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 16:30:16.133845 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129391    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 16:30:16.133845 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129393    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 16:30:16.133845 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129396    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 16:30:16.133845 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129399    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 16:30:16.133845 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129401    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 16:30:16.133845 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129405    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 16:30:16.133845 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129408    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 16:30:16.133845 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129411    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 16:30:16.133845 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129414    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 16:30:16.133845 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129417    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 16:30:16.133845 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129420    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 16:30:16.133845 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129422    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 16:30:16.133845 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129424    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 16:30:16.133845 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129427    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 16:30:16.133845 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129429    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 16:30:16.134293 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129432    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 16:30:16.134293 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129434    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 16:30:16.134293 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129436    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 16:30:16.134293 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129439    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 16:30:16.134293 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129441    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 16:30:16.134293 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129444    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 16:30:16.134293 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129446    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 16:30:16.134293 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129449    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 16:30:16.134293 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129451    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 16:30:16.134293 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129454    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 16:30:16.134293 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129457    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 16:30:16.134293 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129459    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 16:30:16.134293 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129462    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 16:30:16.134293 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129464    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 16:30:16.134293 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129466    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 16:30:16.134293 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129469    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 16:30:16.134293 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129471    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 16:30:16.134293 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129474    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 16:30:16.134293 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129476    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 16:30:16.134751 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129479    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 16:30:16.134751 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129482    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 16:30:16.134751 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129484    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 16:30:16.134751 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129487    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 16:30:16.134751 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129489    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 16:30:16.134751 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129492    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 16:30:16.134751 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129494    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 16:30:16.134751 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129497    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 16:30:16.134751 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129499    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 16:30:16.134751 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129502    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 16:30:16.134751 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129504    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 16:30:16.134751 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129507    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 16:30:16.134751 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129509    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 16:30:16.134751 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129512    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 16:30:16.134751 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129514    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 16:30:16.134751 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129516    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 16:30:16.134751 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129519    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 16:30:16.134751 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129521    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 16:30:16.134751 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129523    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 16:30:16.134751 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129526    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 16:30:16.135306 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129528    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 16:30:16.135306 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129531    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 16:30:16.135306 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129533    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 16:30:16.135306 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129536    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 16:30:16.135306 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129539    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 16:30:16.135306 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129541    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 16:30:16.135306 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129544    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 16:30:16.135306 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:16.129546    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 16:30:16.135306 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.129551    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 16:30:16.135306 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.129672    2557 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 16:30:16.135306 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.132374    2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 16:30:16.135306 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.133366    2557 server.go:1019] "Starting client certificate rotation"
Sep 08 16:30:16.135306 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.133468    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 16:30:16.135306 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.134210    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 16:30:16.135783 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.135770    2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 16:30:16.138794 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.138761    2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 16:30:16.151052 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.151028    2557 log.go:25] "Validated CRI v1 runtime API"
Sep 08 16:30:16.155677 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.155659    2557 log.go:25] "Validated CRI v1 image API"
Sep 08 16:30:16.156595 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.156575    2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 16:30:16.160648 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.160625    2557 fs.go:135] Filesystem UUIDs: map[62c2a735-1562-4a15-a40f-89f416b52652:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a9c1c2e6-74e3-4d7c-b85e-633e1ae5d786:/dev/nvme0n1p3]
Sep 08 16:30:16.160724 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.160646    2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 16:30:16.165798 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.165676    2557 manager.go:216] Machine: {Timestamp:2026-09-08 16:30:16.164415182 +0000 UTC m=+0.327386622 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098826 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26989c33019915729bf76bba1a27cd SystemUUID:ec26989c-3301-9915-729b-f76bba1a27cd BootID:4db5a621-0f86-4c7c-8278-cdc546e9e105 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8c:de:f9:23:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8c:de:f9:23:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:88:8f:f7:98:f6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 16:30:16.165798 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.165788    2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 16:30:16.165931 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.165892    2557 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 16:30:16.167458 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.167442    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 16:30:16.169274 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.169243    2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 16:30:16.169412 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.169278    2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-51.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 16:30:16.170680 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.170668    2557 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 16:30:16.170680 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.170682    2557 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 16:30:16.170796 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.170690    2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 16:30:16.173896 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.173883    2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 16:30:16.175161 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.175150    2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 16:30:16.181019 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.181004    2557 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 16:30:16.181139 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.181129    2557 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 16:30:16.181179 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.181161    2557 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 16:30:16.181179 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.181171    2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 16:30:16.181255 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.181185    2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 16:30:16.181255 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.181196    2557 kubelet.go:398] "Adding apiserver pod source"
Sep 08 16:30:16.181255 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.181205    2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 16:30:16.181608 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.181597    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 16:30:16.181658 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.181624    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 16:30:16.184956 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.184932    2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 16:30:16.187325 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.187309    2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 16:30:16.187400 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.187340    2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 16:30:16.187512 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.187499    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 16:30:16.187555 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.187522    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 16:30:16.187555 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.187532    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 16:30:16.187555 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.187541    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 16:30:16.187555 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.187549    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 16:30:16.187555 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.187555    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 16:30:16.188075 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.188066    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 16:30:16.188075 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.188075    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 16:30:16.188132 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.188084    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 16:30:16.188132 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.188090    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 16:30:16.188132 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.188099    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 16:30:16.188132 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.188107    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 16:30:16.188914 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.188904    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 16:30:16.188914 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.188914    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 16:30:16.192644 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.192614    2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-51.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 16:30:16.192789 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:16.192758    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-51.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 16:30:16.193220 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:16.193191    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 16:30:16.193350 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.193336    2557 server.go:1268] "Started kubelet"
Sep 08 16:30:16.193521 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.193462    2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 16:30:16.193602 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.193505    2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 16:30:16.193602 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.193542    2557 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 16:30:16.194182 ip-10-0-139-51 systemd[1]: Started Kubernetes Kubelet.
Sep 08 16:30:16.195025 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.194672    2557 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 16:30:16.195025 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.194761    2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 16:30:16.198239 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.198216    2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 16:30:16.198318 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.198224    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 16:30:16.198378 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:16.198335    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found"
Sep 08 16:30:16.198753 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:16.197713    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-51.ec2.internal.18d36571be385eda  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-51.ec2.internal,UID:ip-10-0-139-51.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-51.ec2.internal,},FirstTimestamp:2026-09-08 16:30:16.193294042 +0000 UTC m=+0.356265482,LastTimestamp:2026-09-08 16:30:16.193294042 +0000 UTC m=+0.356265482,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-51.ec2.internal,}"
Sep 08 16:30:16.198909 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.198850    2557 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 16:30:16.198909 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.198870    2557 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 16:30:16.199002 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.198984    2557 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 16:30:16.199002 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.198995    2557 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 16:30:16.199108 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.199086    2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 16:30:16.199204 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.199186    2557 factory.go:55] Registering systemd factory
Sep 08 16:30:16.199204 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.199202    2557 factory.go:223] Registration of the systemd container factory successfully
Sep 08 16:30:16.199495 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.199482    2557 factory.go:153] Registering CRI-O factory
Sep 08 16:30:16.199530 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.199499    2557 factory.go:223] Registration of the crio container factory successfully
Sep 08 16:30:16.199566 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.199540    2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 16:30:16.199566 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.199560    2557 factory.go:103] Registering Raw factory
Sep 08 16:30:16.199638 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.199574    2557 manager.go:1196] Started watching for new ooms in manager
Sep 08 16:30:16.199819 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:16.199763    2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 16:30:16.200025 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.200001    2557 manager.go:349] Starting recovery of all containers
Sep 08 16:30:16.202968 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.202941    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tr6k4"
Sep 08 16:30:16.210138 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.210118    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tr6k4"
Sep 08 16:30:16.211085 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:16.211060    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 08 16:30:16.211185 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:16.211143    2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-51.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 08 16:30:16.211386 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.211368    2557 manager.go:354] Recovery completed
Sep 08 16:30:16.217919 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.217905    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:30:16.219661 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.219646    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:30:16.219713 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.219677    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:30:16.219713 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.219689    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:30:16.220222 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.220208    2557 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 16:30:16.220222 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.220220    2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 16:30:16.220331 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.220234    2557 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 16:30:16.220331 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.220291    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:30:16.220979 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.220961    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:30:16.221055 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.220992    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:30:16.221055 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.221003    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:30:16.221645 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.221629    2557 policy_none.go:49] "None policy: Start"
Sep 08 16:30:16.221712 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.221650    2557 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 16:30:16.221712 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.221663    2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 16:30:16.221775 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:16.221677    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-51.ec2.internal.18d36571bfcab560  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-51.ec2.internal,UID:ip-10-0-139-51.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-51.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-51.ec2.internal,},FirstTimestamp:2026-09-08 16:30:16.219661664 +0000 UTC m=+0.382633102,LastTimestamp:2026-09-08 16:30:16.219661664 +0000 UTC m=+0.382633102,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-51.ec2.internal,}"
Sep 08 16:30:16.222848 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.222822    2557 policy_none.go:47] "Start"
Sep 08 16:30:16.265449 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.265433    2557 manager.go:337] "Starting Device Plugin manager"
Sep 08 16:30:16.274513 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:16.265471    2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 16:30:16.274513 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.265482    2557 server.go:87] "Starting device plugin registration server"
Sep 08 16:30:16.274513 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.265746    2557 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 16:30:16.274513 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.265759    2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 16:30:16.274513 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.265863    2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 16:30:16.274513 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.265932    2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 16:30:16.274513 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.265940    2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 16:30:16.274513 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:16.266476    2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 16:30:16.274513 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:16.266522    2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-51.ec2.internal\" not found"
Sep 08 16:30:16.337601 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.337561    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 16:30:16.339044 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.339029    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 16:30:16.339122 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.339080    2557 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 16:30:16.339122 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.339104    2557 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 16:30:16.339192 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:16.339147    2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 16:30:16.342052 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.342031    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 16:30:16.366385 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.366317    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:30:16.367250 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.367235    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:30:16.367326 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.367266    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:30:16.367326 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.367277    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:30:16.367326 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.367299    2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-51.ec2.internal"
Sep 08 16:30:16.376207 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.376188    2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-51.ec2.internal"
Sep 08 16:30:16.376260 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:16.376218    2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-51.ec2.internal\": node \"ip-10-0-139-51.ec2.internal\" not found"
Sep 08 16:30:16.397130 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:16.397099    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found"
Sep 08 16:30:16.440075 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.440050    2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal"]
Sep 08 16:30:16.440140 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.440121    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:30:16.441674 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.441660    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:30:16.441739 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.441692    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:30:16.441739 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.441707    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:30:16.442923 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.442911    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:30:16.443051 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.443039    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal"
Sep 08 16:30:16.443085 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.443068    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:30:16.443709 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.443693    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:30:16.443770 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.443708    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:30:16.443770 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.443724    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:30:16.443770 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.443732    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:30:16.443770 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.443737    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:30:16.443770 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.443747    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:30:16.444900 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.444887    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal"
Sep 08 16:30:16.444956 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.444912    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:30:16.446037 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.446021    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:30:16.446134 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.446045    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:30:16.446134 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.446060    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:30:16.468629 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:16.468605    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-51.ec2.internal\" not found" node="ip-10-0-139-51.ec2.internal"
Sep 08 16:30:16.472118 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:16.472102    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-51.ec2.internal\" not found" node="ip-10-0-139-51.ec2.internal"
Sep 08 16:30:16.497632 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:16.497611    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found"
Sep 08 16:30:16.500913 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.500897    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/555e95963eeb3a1ffd9bfc39675ae3f8-config\") pod \"kube-apiserver-proxy-ip-10-0-139-51.ec2.internal\" (UID: \"555e95963eeb3a1ffd9bfc39675ae3f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal"
Sep 08 16:30:16.500969 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.500921    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/179f538010bbe319e093cd4c1c6661f9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal\" (UID: \"179f538010bbe319e093cd4c1c6661f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal"
Sep 08 16:30:16.500969 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.500942    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/179f538010bbe319e093cd4c1c6661f9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal\" (UID: \"179f538010bbe319e093cd4c1c6661f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal"
Sep 08 16:30:16.598061 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:16.598030    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found"
Sep 08 16:30:16.601286 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.601266    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/179f538010bbe319e093cd4c1c6661f9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal\" (UID: \"179f538010bbe319e093cd4c1c6661f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal"
Sep 08 16:30:16.601348 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.601296    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/179f538010bbe319e093cd4c1c6661f9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal\" (UID: \"179f538010bbe319e093cd4c1c6661f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal"
Sep 08 16:30:16.601348 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.601314    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/555e95963eeb3a1ffd9bfc39675ae3f8-config\") pod \"kube-apiserver-proxy-ip-10-0-139-51.ec2.internal\" (UID: \"555e95963eeb3a1ffd9bfc39675ae3f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal"
Sep 08 16:30:16.601438 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.601382    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/179f538010bbe319e093cd4c1c6661f9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal\" (UID: \"179f538010bbe319e093cd4c1c6661f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal"
Sep 08 16:30:16.601438 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.601387    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/555e95963eeb3a1ffd9bfc39675ae3f8-config\") pod \"kube-apiserver-proxy-ip-10-0-139-51.ec2.internal\" (UID: \"555e95963eeb3a1ffd9bfc39675ae3f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal"
Sep 08 16:30:16.601515 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.601382    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/179f538010bbe319e093cd4c1c6661f9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal\" (UID: \"179f538010bbe319e093cd4c1c6661f9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal"
Sep 08 16:30:16.699117 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:16.699088    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found"
Sep 08 16:30:16.771489 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.771465    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal"
Sep 08 16:30:16.774208 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:16.774184    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal"
Sep 08 16:30:16.799926 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:16.799748    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found"
Sep 08 16:30:16.900224 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:16.900187    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found"
Sep 08 16:30:17.000696 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:17.000603    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found"
Sep 08 16:30:17.101346 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:17.101313    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found"
Sep 08 16:30:17.133618 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:17.133600    2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 16:30:17.134007 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:17.133771    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 16:30:17.160839 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:17.160815    2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 16:30:17.162497 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:17.162476    2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 16:30:17.202154 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:17.202129    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found"
Sep 08 16:30:17.213424 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:17.213391    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 16:25:16 +0000 UTC" deadline="2028-06-19 04:55:22.504671326 +0000 UTC"
Sep 08 16:30:17.213424 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:17.213418    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15588h25m5.291256745s"
Sep 08 16:30:17.303013 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:17.302948    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found"
Sep 08 16:30:17.403968 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:17.403937    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-51.ec2.internal\" not found"
Sep 08 16:30:17.414321 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:17.414296    2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 16:30:17.465885 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:17.465845    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179f538010bbe319e093cd4c1c6661f9.slice/crio-3c2f87f3d17bd053bd378a0cbf353f584335dd3fb9af3313a4611155f7f69127 WatchSource:0}: Error finding container 3c2f87f3d17bd053bd378a0cbf353f584335dd3fb9af3313a4611155f7f69127: Status 404 returned error can't find the container with id 3c2f87f3d17bd053bd378a0cbf353f584335dd3fb9af3313a4611155f7f69127
Sep 08 16:30:17.470600 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:17.470582    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 16:30:17.499667 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:17.499633    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal"
Sep 08 16:30:17.517906 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:17.517881    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 16:30:17.519518 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:17.519504    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal"
Sep 08 16:30:17.531249 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:17.531230    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 16:30:17.986656 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:17.986620    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 16:30:18.051146 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:18.051114    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555e95963eeb3a1ffd9bfc39675ae3f8.slice/crio-7b2c1c86f80d8b0b8c3a8ffd93105d46f2383fa5d515382642465c55036de07d WatchSource:0}: Error finding container 7b2c1c86f80d8b0b8c3a8ffd93105d46f2383fa5d515382642465c55036de07d: Status 404 returned error can't find the container with id 7b2c1c86f80d8b0b8c3a8ffd93105d46f2383fa5d515382642465c55036de07d
Sep 08 16:30:18.185626 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.185576    2557 apiserver.go:52] "Watching apiserver"
Sep 08 16:30:18.193431 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.193404    2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 16:30:18.193783 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.193757    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-r4kbv","kube-system/global-pull-secret-syncer-g67mm","kube-system/konnectivity-agent-pqhsk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6","openshift-image-registry/node-ca-m4n88","openshift-multus/multus-additional-cni-plugins-62zlz","openshift-ovn-kubernetes/ovnkube-node-mfqgd","kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal","openshift-cluster-node-tuning-operator/tuned-dxt9v","openshift-dns/node-resolver-rj64h","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal","openshift-multus/multus-q8kt4","openshift-multus/network-metrics-daemon-fm6rg","openshift-network-diagnostics/network-check-target-mz7tj"]
Sep 08 16:30:18.196838 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.196817    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.199082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.199060    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 16:30:18.199183 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.199112    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:18.199225 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:18.199187    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fm6rg" podUID="48a3c156-d793-4586-9ef9-c6770eb982d0"
Sep 08 16:30:18.200855 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.200837    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 16:30:18.200985 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.200907    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 16:30:18.201288 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.201260    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 16:30:18.201492 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.201477    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pqhsk"
Sep 08 16:30:18.202316 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.202296    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 16:30:18.203023 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.203005    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 16:30:18.203174 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.203158    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hkw95\""
Sep 08 16:30:18.203932 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.203914    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6"
Sep 08 16:30:18.204001 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.203932    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 16:30:18.204970 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.204949    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 16:30:18.205067 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.204973    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 16:30:18.205067 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.204973    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rcwqc\""
Sep 08 16:30:18.206350 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.206334    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m4n88"
Sep 08 16:30:18.207975 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.207956    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 16:30:18.208250 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.208235    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 16:30:18.208657 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.208639    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 16:30:18.208943 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.208921    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-62zlz"
Sep 08 16:30:18.209871 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.209855    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-run-systemd\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.209974 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.209879    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-etc-openvswitch\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.209974 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.209897    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-log-socket\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.209974 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.209920    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/41e9b515-3988-4240-a220-e3325ecfbcee-ovnkube-script-lib\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.209974 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.209945    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j85pv\" (UniqueName: \"kubernetes.io/projected/41e9b515-3988-4240-a220-e3325ecfbcee-kube-api-access-j85pv\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.210115 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.209999    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7w4w\" (UniqueName: \"kubernetes.io/projected/48a3c156-d793-4586-9ef9-c6770eb982d0-kube-api-access-w7w4w\") pod \"network-metrics-daemon-fm6rg\" (UID: \"48a3c156-d793-4586-9ef9-c6770eb982d0\") " pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:18.210115 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.210051    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bd8c514b-7ca3-43e4-b87c-8a811c04efe8-agent-certs\") pod \"konnectivity-agent-pqhsk\" (UID: \"bd8c514b-7ca3-43e4-b87c-8a811c04efe8\") " pod="kube-system/konnectivity-agent-pqhsk"
Sep 08 16:30:18.210115 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.210069    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bd8c514b-7ca3-43e4-b87c-8a811c04efe8-konnectivity-ca\") pod \"konnectivity-agent-pqhsk\" (UID: \"bd8c514b-7ca3-43e4-b87c-8a811c04efe8\") " pod="kube-system/konnectivity-agent-pqhsk"
Sep 08 16:30:18.210115 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.210084    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-var-lib-openvswitch\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.210115 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.210103    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-host-run-ovn-kubernetes\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.210287 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.210140    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-host-cni-bin\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.210287 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.210164    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-host-cni-netd\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.210287 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.210186    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-host-slash\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.210287 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.210203    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-run-ovn\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.210287 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.210257    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.210438 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.210289    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48a3c156-d793-4586-9ef9-c6770eb982d0-metrics-certs\") pod \"network-metrics-daemon-fm6rg\" (UID: \"48a3c156-d793-4586-9ef9-c6770eb982d0\") " pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:18.210438 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.210320    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-host-kubelet\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.210438 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.210347    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-systemd-units\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.210438 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.210370    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-run-openvswitch\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.210438 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.210394    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-node-log\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.210587 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.210436    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/41e9b515-3988-4240-a220-e3325ecfbcee-ovnkube-config\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.210587 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.210476    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/41e9b515-3988-4240-a220-e3325ecfbcee-env-overrides\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.210587 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.210503    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/41e9b515-3988-4240-a220-e3325ecfbcee-ovn-node-metrics-cert\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.210587 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.210528    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-host-run-netns\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.211465 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.211448    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.213314 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.213291    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 16:30:18.213314 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.213306    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xnw89\""
Sep 08 16:30:18.213465 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.213298    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 16:30:18.213465 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.213457    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 16:30:18.213638 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.213624    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 16:30:18.213702 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.213645    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r2w9c\""
Sep 08 16:30:18.213799 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.213781    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r4kbv"
Sep 08 16:30:18.214056 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.213927    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 16:30:18.214056 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.213971    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 16:30:18.214525 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.214457    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 16:30:18.214619 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.214570    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cvhxf\""
Sep 08 16:30:18.214673 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.214639    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 16:30:18.214757 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.214732    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bl5d2\""
Sep 08 16:30:18.215197 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.215180    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 16:30:18.216505 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.216147    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 16:30:18.218935 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.216834    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 16:30:18.218935 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.216879    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8dnfs\""
Sep 08 16:30:18.218935 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.216958    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 16:30:18.218935 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.217212    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 16:30:18.218935 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.217212    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 16:30:18.218935 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.217910    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:18.218935 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:18.218006    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g67mm" podUID="0abcdd93-6ae1-4e51-a19b-911c06152822"
Sep 08 16:30:18.220787 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.220771    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rj64h"
Sep 08 16:30:18.223103 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.223087    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:18.223199 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:18.223161    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz7tj" podUID="50d7b024-55d7-4077-8584-9657bf3008b5"
Sep 08 16:30:18.225517 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.225497    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.226615 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.226593    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 16:30:18.226707 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.226656    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 16:30:18.227923 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.227902    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fgdln\""
Sep 08 16:30:18.228013 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.227952    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 16:30:18.228080 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.228061    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ps52j\""
Sep 08 16:30:18.255521 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.255498    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4p77j"
Sep 08 16:30:18.265455 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.265434    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4p77j"
Sep 08 16:30:18.300328 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.300307    2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 16:30:18.310823 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.310801    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0abcdd93-6ae1-4e51-a19b-911c06152822-original-pull-secret\") pod \"global-pull-secret-syncer-g67mm\" (UID: \"0abcdd93-6ae1-4e51-a19b-911c06152822\") " pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:18.310958 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.310830    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-multus-conf-dir\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.310958 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.310872    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d1f5282-a130-4c07-86b0-c70f99879223-device-dir\") pod \"aws-ebs-csi-driver-node-nrmb6\" (UID: \"3d1f5282-a130-4c07-86b0-c70f99879223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6"
Sep 08 16:30:18.310958 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.310891    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d5c4bf16-72bf-41a9-bc8a-58cc50aa0cba-host-slash\") pod \"iptables-alerter-r4kbv\" (UID: \"d5c4bf16-72bf-41a9-bc8a-58cc50aa0cba\") " pod="openshift-network-operator/iptables-alerter-r4kbv"
Sep 08 16:30:18.310958 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.310930    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-etc-sysctl-d\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.311164 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.310959    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-etc-sysctl-conf\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.311164 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.310991    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-host-run-netns\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.311164 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311018    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhbqb\" (UniqueName: \"kubernetes.io/projected/3d1f5282-a130-4c07-86b0-c70f99879223-kube-api-access-bhbqb\") pod \"aws-ebs-csi-driver-node-nrmb6\" (UID: \"3d1f5282-a130-4c07-86b0-c70f99879223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6"
Sep 08 16:30:18.311164 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311055    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-host-run-netns\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.311164 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311072    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/10038ceb-d586-4fb2-99e2-dc326d9daa8c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-62zlz\" (UID: \"10038ceb-d586-4fb2-99e2-dc326d9daa8c\") " pod="openshift-multus/multus-additional-cni-plugins-62zlz"
Sep 08 16:30:18.311164 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311108    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bd8c514b-7ca3-43e4-b87c-8a811c04efe8-konnectivity-ca\") pod \"konnectivity-agent-pqhsk\" (UID: \"bd8c514b-7ca3-43e4-b87c-8a811c04efe8\") " pod="kube-system/konnectivity-agent-pqhsk"
Sep 08 16:30:18.311386 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311197    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-host-cni-bin\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.311386 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311224    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-host-cni-netd\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.311386 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311242    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-multus-socket-dir-parent\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.311386 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311261    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d1f5282-a130-4c07-86b0-c70f99879223-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nrmb6\" (UID: \"3d1f5282-a130-4c07-86b0-c70f99879223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6"
Sep 08 16:30:18.311386 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311278    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvcz8\" (UniqueName: \"kubernetes.io/projected/50d7b024-55d7-4077-8584-9657bf3008b5-kube-api-access-hvcz8\") pod \"network-check-target-mz7tj\" (UID: \"50d7b024-55d7-4077-8584-9657bf3008b5\") " pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:18.311386 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311294    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b925da05-aa52-48c7-8dc0-839d562e1d01-etc-tuned\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.311386 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311304    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-host-cni-netd\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.311386 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311313    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-host-cni-bin\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.311386 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311310    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-run-ovn\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.311386 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311352    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.311386 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311369    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-run-ovn\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.311761 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311402    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g756r\" (UniqueName: \"kubernetes.io/projected/ff68e8af-fa2a-4b97-a923-3db24f91d137-kube-api-access-g756r\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.311761 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311435    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.311761 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311483    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/41e9b515-3988-4240-a220-e3325ecfbcee-ovnkube-config\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.311761 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311507    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/41e9b515-3988-4240-a220-e3325ecfbcee-env-overrides\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.311761 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311523    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/41e9b515-3988-4240-a220-e3325ecfbcee-ovn-node-metrics-cert\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.311761 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311542    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph7sw\" (UniqueName: \"kubernetes.io/projected/ad2a7c50-5f92-449d-a650-ed26da4359b7-kube-api-access-ph7sw\") pod \"node-resolver-rj64h\" (UID: \"ad2a7c50-5f92-449d-a650-ed26da4359b7\") " pod="openshift-dns/node-resolver-rj64h"
Sep 08 16:30:18.311761 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311560    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-system-cni-dir\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.311761 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311582    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/afe27658-50c9-47a8-884a-8ddc05f375ed-host\") pod \"node-ca-m4n88\" (UID: \"afe27658-50c9-47a8-884a-8ddc05f375ed\") " pod="openshift-image-registry/node-ca-m4n88"
Sep 08 16:30:18.311761 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311625    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-etc-sysconfig\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.311761 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311670    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2whdh\" (UniqueName: \"kubernetes.io/projected/afe27658-50c9-47a8-884a-8ddc05f375ed-kube-api-access-2whdh\") pod \"node-ca-m4n88\" (UID: \"afe27658-50c9-47a8-884a-8ddc05f375ed\") " pod="openshift-image-registry/node-ca-m4n88"
Sep 08 16:30:18.311761 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311705    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-etc-openvswitch\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.311761 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311728    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-hostroot\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.311761 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311749    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-etc-modprobe-d\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.312330 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311774    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b925da05-aa52-48c7-8dc0-839d562e1d01-tmp\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.312330 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311802    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt2vb\" (UniqueName: \"kubernetes.io/projected/b925da05-aa52-48c7-8dc0-839d562e1d01-kube-api-access-bt2vb\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.312330 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311805    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-etc-openvswitch\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.312330 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311853    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-host-run-ovn-kubernetes\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.312330 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311887    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-multus-cni-dir\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.312330 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311911    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-host-run-ovn-kubernetes\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.312330 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311912    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10038ceb-d586-4fb2-99e2-dc326d9daa8c-cni-binary-copy\") pod \"multus-additional-cni-plugins-62zlz\" (UID: \"10038ceb-d586-4fb2-99e2-dc326d9daa8c\") " pod="openshift-multus/multus-additional-cni-plugins-62zlz"
Sep 08 16:30:18.312330 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311946    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-run\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.312330 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311968    2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 16:30:18.312330 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.311970    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-host\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.312330 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312025    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-host-slash\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.312330 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312049    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d1f5282-a130-4c07-86b0-c70f99879223-sys-fs\") pod \"aws-ebs-csi-driver-node-nrmb6\" (UID: \"3d1f5282-a130-4c07-86b0-c70f99879223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6"
Sep 08 16:30:18.312330 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312078    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-systemd-units\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.312330 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312081    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/41e9b515-3988-4240-a220-e3325ecfbcee-env-overrides\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.312330 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312105    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-run-openvswitch\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.312330 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312141    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-node-log\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.312330 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312145    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-host-slash\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.313024 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312164    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-systemd-units\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.313024 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312172    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-run-openvswitch\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.313024 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312196    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/41e9b515-3988-4240-a220-e3325ecfbcee-ovnkube-config\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.313024 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312210    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff68e8af-fa2a-4b97-a923-3db24f91d137-cni-binary-copy\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.313024 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312226    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bd8c514b-7ca3-43e4-b87c-8a811c04efe8-konnectivity-ca\") pod \"konnectivity-agent-pqhsk\" (UID: \"bd8c514b-7ca3-43e4-b87c-8a811c04efe8\") " pod="kube-system/konnectivity-agent-pqhsk"
Sep 08 16:30:18.313024 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312238    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-node-log\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.313024 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312282    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-host-var-lib-kubelet\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.313024 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312315    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d1f5282-a130-4c07-86b0-c70f99879223-etc-selinux\") pod \"aws-ebs-csi-driver-node-nrmb6\" (UID: \"3d1f5282-a130-4c07-86b0-c70f99879223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6"
Sep 08 16:30:18.313024 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312351    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6xvx\" (UniqueName: \"kubernetes.io/projected/10038ceb-d586-4fb2-99e2-dc326d9daa8c-kube-api-access-k6xvx\") pod \"multus-additional-cni-plugins-62zlz\" (UID: \"10038ceb-d586-4fb2-99e2-dc326d9daa8c\") " pod="openshift-multus/multus-additional-cni-plugins-62zlz"
Sep 08 16:30:18.313024 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312377    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-etc-kubernetes\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.313024 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312398    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-log-socket\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.313024 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312415    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7w4w\" (UniqueName: \"kubernetes.io/projected/48a3c156-d793-4586-9ef9-c6770eb982d0-kube-api-access-w7w4w\") pod \"network-metrics-daemon-fm6rg\" (UID: \"48a3c156-d793-4586-9ef9-c6770eb982d0\") " pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:18.313024 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312461    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-log-socket\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.313024 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312473    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/afe27658-50c9-47a8-884a-8ddc05f375ed-serviceca\") pod \"node-ca-m4n88\" (UID: \"afe27658-50c9-47a8-884a-8ddc05f375ed\") " pod="openshift-image-registry/node-ca-m4n88"
Sep 08 16:30:18.313024 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312498    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10038ceb-d586-4fb2-99e2-dc326d9daa8c-system-cni-dir\") pod \"multus-additional-cni-plugins-62zlz\" (UID: \"10038ceb-d586-4fb2-99e2-dc326d9daa8c\") " pod="openshift-multus/multus-additional-cni-plugins-62zlz"
Sep 08 16:30:18.313024 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312516    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-var-lib-kubelet\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.313024 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312534    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bd8c514b-7ca3-43e4-b87c-8a811c04efe8-agent-certs\") pod \"konnectivity-agent-pqhsk\" (UID: \"bd8c514b-7ca3-43e4-b87c-8a811c04efe8\") " pod="kube-system/konnectivity-agent-pqhsk"
Sep 08 16:30:18.313844 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312560    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-var-lib-openvswitch\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.313844 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312577    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48a3c156-d793-4586-9ef9-c6770eb982d0-metrics-certs\") pod \"network-metrics-daemon-fm6rg\" (UID: \"48a3c156-d793-4586-9ef9-c6770eb982d0\") " pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:18.313844 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312592    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ad2a7c50-5f92-449d-a650-ed26da4359b7-hosts-file\") pod \"node-resolver-rj64h\" (UID: \"ad2a7c50-5f92-449d-a650-ed26da4359b7\") " pod="openshift-dns/node-resolver-rj64h"
Sep 08 16:30:18.313844 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312607    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-cnibin\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.313844 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312632    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-host-run-multus-certs\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.313844 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:18.312702    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:30:18.313844 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312763    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-var-lib-openvswitch\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.313844 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312764    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-etc-systemd\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.313844 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312801    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-host-run-k8s-cni-cncf-io\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.313844 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:18.312841    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48a3c156-d793-4586-9ef9-c6770eb982d0-metrics-certs podName:48a3c156-d793-4586-9ef9-c6770eb982d0 nodeName:}" failed. No retries permitted until 2026-09-08 16:30:18.812795233 +0000 UTC m=+2.975766676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48a3c156-d793-4586-9ef9-c6770eb982d0-metrics-certs") pod "network-metrics-daemon-fm6rg" (UID: "48a3c156-d793-4586-9ef9-c6770eb982d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:30:18.313844 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312902    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0abcdd93-6ae1-4e51-a19b-911c06152822-kubelet-config\") pod \"global-pull-secret-syncer-g67mm\" (UID: \"0abcdd93-6ae1-4e51-a19b-911c06152822\") " pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:18.313844 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312961    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-host-kubelet\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.313844 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.312990    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j85pv\" (UniqueName: \"kubernetes.io/projected/41e9b515-3988-4240-a220-e3325ecfbcee-kube-api-access-j85pv\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.313844 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.313019    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d1f5282-a130-4c07-86b0-c70f99879223-registration-dir\") pod \"aws-ebs-csi-driver-node-nrmb6\" (UID: \"3d1f5282-a130-4c07-86b0-c70f99879223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6"
Sep 08 16:30:18.313844 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.313033    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-host-kubelet\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.313844 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.313042    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xf79\" (UniqueName: \"kubernetes.io/projected/d5c4bf16-72bf-41a9-bc8a-58cc50aa0cba-kube-api-access-5xf79\") pod \"iptables-alerter-r4kbv\" (UID: \"d5c4bf16-72bf-41a9-bc8a-58cc50aa0cba\") " pod="openshift-network-operator/iptables-alerter-r4kbv"
Sep 08 16:30:18.314443 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.313115    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10038ceb-d586-4fb2-99e2-dc326d9daa8c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-62zlz\" (UID: \"10038ceb-d586-4fb2-99e2-dc326d9daa8c\") " pod="openshift-multus/multus-additional-cni-plugins-62zlz"
Sep 08 16:30:18.314443 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.313156    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/10038ceb-d586-4fb2-99e2-dc326d9daa8c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-62zlz\" (UID: \"10038ceb-d586-4fb2-99e2-dc326d9daa8c\") " pod="openshift-multus/multus-additional-cni-plugins-62zlz"
Sep 08 16:30:18.314443 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.313184    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-sys\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.314443 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.313209    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-run-systemd\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.314443 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.313235    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/41e9b515-3988-4240-a220-e3325ecfbcee-ovnkube-script-lib\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.314443 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.313261    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-host-var-lib-cni-bin\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.314443 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.313306    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-host-var-lib-cni-multus\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.314443 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.313458    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d5c4bf16-72bf-41a9-bc8a-58cc50aa0cba-iptables-alerter-script\") pod \"iptables-alerter-r4kbv\" (UID: \"d5c4bf16-72bf-41a9-bc8a-58cc50aa0cba\") " pod="openshift-network-operator/iptables-alerter-r4kbv"
Sep 08 16:30:18.314443 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.313338    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/41e9b515-3988-4240-a220-e3325ecfbcee-run-systemd\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.314443 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.313563    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-lib-modules\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.314443 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.313589    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad2a7c50-5f92-449d-a650-ed26da4359b7-tmp-dir\") pod \"node-resolver-rj64h\" (UID: \"ad2a7c50-5f92-449d-a650-ed26da4359b7\") " pod="openshift-dns/node-resolver-rj64h"
Sep 08 16:30:18.314443 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.313611    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-host-run-netns\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.314443 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.313631    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ff68e8af-fa2a-4b97-a923-3db24f91d137-multus-daemon-config\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.314443 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.313652    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10038ceb-d586-4fb2-99e2-dc326d9daa8c-cnibin\") pod \"multus-additional-cni-plugins-62zlz\" (UID: \"10038ceb-d586-4fb2-99e2-dc326d9daa8c\") " pod="openshift-multus/multus-additional-cni-plugins-62zlz"
Sep 08 16:30:18.314443 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.313676    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-etc-kubernetes\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.314443 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.313699    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d1f5282-a130-4c07-86b0-c70f99879223-socket-dir\") pod \"aws-ebs-csi-driver-node-nrmb6\" (UID: \"3d1f5282-a130-4c07-86b0-c70f99879223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6"
Sep 08 16:30:18.314443 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.313752    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0abcdd93-6ae1-4e51-a19b-911c06152822-dbus\") pod \"global-pull-secret-syncer-g67mm\" (UID: \"0abcdd93-6ae1-4e51-a19b-911c06152822\") " pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:18.314965 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.313902    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/41e9b515-3988-4240-a220-e3325ecfbcee-ovnkube-script-lib\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.315827 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.315802    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/41e9b515-3988-4240-a220-e3325ecfbcee-ovn-node-metrics-cert\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.315947 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.315929    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bd8c514b-7ca3-43e4-b87c-8a811c04efe8-agent-certs\") pod \"konnectivity-agent-pqhsk\" (UID: \"bd8c514b-7ca3-43e4-b87c-8a811c04efe8\") " pod="kube-system/konnectivity-agent-pqhsk"
Sep 08 16:30:18.335246 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.335216    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7w4w\" (UniqueName: \"kubernetes.io/projected/48a3c156-d793-4586-9ef9-c6770eb982d0-kube-api-access-w7w4w\") pod \"network-metrics-daemon-fm6rg\" (UID: \"48a3c156-d793-4586-9ef9-c6770eb982d0\") " pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:18.339571 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.339542    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j85pv\" (UniqueName: \"kubernetes.io/projected/41e9b515-3988-4240-a220-e3325ecfbcee-kube-api-access-j85pv\") pod \"ovnkube-node-mfqgd\" (UID: \"41e9b515-3988-4240-a220-e3325ecfbcee\") " pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.343526 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.343480    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal" event={"ID":"555e95963eeb3a1ffd9bfc39675ae3f8","Type":"ContainerStarted","Data":"7b2c1c86f80d8b0b8c3a8ffd93105d46f2383fa5d515382642465c55036de07d"}
Sep 08 16:30:18.347298 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.347264    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" event={"ID":"179f538010bbe319e093cd4c1c6661f9","Type":"ContainerStarted","Data":"3c2f87f3d17bd053bd378a0cbf353f584335dd3fb9af3313a4611155f7f69127"}
Sep 08 16:30:18.414715 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.414678    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-multus-conf-dir\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.414715 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.414715    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d1f5282-a130-4c07-86b0-c70f99879223-device-dir\") pod \"aws-ebs-csi-driver-node-nrmb6\" (UID: \"3d1f5282-a130-4c07-86b0-c70f99879223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6"
Sep 08 16:30:18.415000 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.414733    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d5c4bf16-72bf-41a9-bc8a-58cc50aa0cba-host-slash\") pod \"iptables-alerter-r4kbv\" (UID: \"d5c4bf16-72bf-41a9-bc8a-58cc50aa0cba\") " pod="openshift-network-operator/iptables-alerter-r4kbv"
Sep 08 16:30:18.415000 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.414760    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-etc-sysctl-d\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.415000 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.414786    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-etc-sysctl-conf\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.415000 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.414783    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-multus-conf-dir\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.415000 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.414815    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhbqb\" (UniqueName: \"kubernetes.io/projected/3d1f5282-a130-4c07-86b0-c70f99879223-kube-api-access-bhbqb\") pod \"aws-ebs-csi-driver-node-nrmb6\" (UID: \"3d1f5282-a130-4c07-86b0-c70f99879223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6"
Sep 08 16:30:18.415000 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.414842    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/10038ceb-d586-4fb2-99e2-dc326d9daa8c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-62zlz\" (UID: \"10038ceb-d586-4fb2-99e2-dc326d9daa8c\") " pod="openshift-multus/multus-additional-cni-plugins-62zlz"
Sep 08 16:30:18.415000 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.414819    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d1f5282-a130-4c07-86b0-c70f99879223-device-dir\") pod \"aws-ebs-csi-driver-node-nrmb6\" (UID: \"3d1f5282-a130-4c07-86b0-c70f99879223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6"
Sep 08 16:30:18.415000 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.414869    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d5c4bf16-72bf-41a9-bc8a-58cc50aa0cba-host-slash\") pod \"iptables-alerter-r4kbv\" (UID: \"d5c4bf16-72bf-41a9-bc8a-58cc50aa0cba\") " pod="openshift-network-operator/iptables-alerter-r4kbv"
Sep 08 16:30:18.415000 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.414894    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-multus-socket-dir-parent\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.415000 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.414922    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d1f5282-a130-4c07-86b0-c70f99879223-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nrmb6\" (UID: \"3d1f5282-a130-4c07-86b0-c70f99879223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6"
Sep 08 16:30:18.415000 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.414946    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvcz8\" (UniqueName: \"kubernetes.io/projected/50d7b024-55d7-4077-8584-9657bf3008b5-kube-api-access-hvcz8\") pod \"network-check-target-mz7tj\" (UID: \"50d7b024-55d7-4077-8584-9657bf3008b5\") " pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:18.415000 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.414968    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b925da05-aa52-48c7-8dc0-839d562e1d01-etc-tuned\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.415000 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.414978    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-etc-sysctl-conf\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.415000 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.414979    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-etc-sysctl-d\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.415000 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.414995    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g756r\" (UniqueName: \"kubernetes.io/projected/ff68e8af-fa2a-4b97-a923-3db24f91d137-kube-api-access-g756r\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.415800 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415023    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph7sw\" (UniqueName: \"kubernetes.io/projected/ad2a7c50-5f92-449d-a650-ed26da4359b7-kube-api-access-ph7sw\") pod \"node-resolver-rj64h\" (UID: \"ad2a7c50-5f92-449d-a650-ed26da4359b7\") " pod="openshift-dns/node-resolver-rj64h"
Sep 08 16:30:18.415800 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415046    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-system-cni-dir\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.415800 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415053    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d1f5282-a130-4c07-86b0-c70f99879223-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nrmb6\" (UID: \"3d1f5282-a130-4c07-86b0-c70f99879223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6"
Sep 08 16:30:18.415800 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415071    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/afe27658-50c9-47a8-884a-8ddc05f375ed-host\") pod \"node-ca-m4n88\" (UID: \"afe27658-50c9-47a8-884a-8ddc05f375ed\") " pod="openshift-image-registry/node-ca-m4n88"
Sep 08 16:30:18.415800 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415094    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-etc-sysconfig\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.415800 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415118    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2whdh\" (UniqueName: \"kubernetes.io/projected/afe27658-50c9-47a8-884a-8ddc05f375ed-kube-api-access-2whdh\") pod \"node-ca-m4n88\" (UID: \"afe27658-50c9-47a8-884a-8ddc05f375ed\") " pod="openshift-image-registry/node-ca-m4n88"
Sep 08 16:30:18.415800 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415144    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-hostroot\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.415800 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415169    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-etc-modprobe-d\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.415800 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415178    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/afe27658-50c9-47a8-884a-8ddc05f375ed-host\") pod \"node-ca-m4n88\" (UID: \"afe27658-50c9-47a8-884a-8ddc05f375ed\") " pod="openshift-image-registry/node-ca-m4n88"
Sep 08 16:30:18.415800 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415195    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b925da05-aa52-48c7-8dc0-839d562e1d01-tmp\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.415800 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415022    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-multus-socket-dir-parent\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.415800 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415233    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt2vb\" (UniqueName: \"kubernetes.io/projected/b925da05-aa52-48c7-8dc0-839d562e1d01-kube-api-access-bt2vb\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.415800 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415268    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-multus-cni-dir\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.415800 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415293    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10038ceb-d586-4fb2-99e2-dc326d9daa8c-cni-binary-copy\") pod \"multus-additional-cni-plugins-62zlz\" (UID: \"10038ceb-d586-4fb2-99e2-dc326d9daa8c\") " pod="openshift-multus/multus-additional-cni-plugins-62zlz"
Sep 08 16:30:18.415800 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415318    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-run\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.415800 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415343    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-host\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.415800 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415371    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d1f5282-a130-4c07-86b0-c70f99879223-sys-fs\") pod \"aws-ebs-csi-driver-node-nrmb6\" (UID: \"3d1f5282-a130-4c07-86b0-c70f99879223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6"
Sep 08 16:30:18.415800 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415377    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-etc-sysconfig\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.416703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415397    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff68e8af-fa2a-4b97-a923-3db24f91d137-cni-binary-copy\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.416703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415423    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-host-var-lib-kubelet\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.416703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415430    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-hostroot\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.416703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415448    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d1f5282-a130-4c07-86b0-c70f99879223-etc-selinux\") pod \"aws-ebs-csi-driver-node-nrmb6\" (UID: \"3d1f5282-a130-4c07-86b0-c70f99879223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6"
Sep 08 16:30:18.416703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415473    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6xvx\" (UniqueName: \"kubernetes.io/projected/10038ceb-d586-4fb2-99e2-dc326d9daa8c-kube-api-access-k6xvx\") pod \"multus-additional-cni-plugins-62zlz\" (UID: \"10038ceb-d586-4fb2-99e2-dc326d9daa8c\") " pod="openshift-multus/multus-additional-cni-plugins-62zlz"
Sep 08 16:30:18.416703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415489    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-etc-modprobe-d\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.416703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415502    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-etc-kubernetes\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.416703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415528    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/afe27658-50c9-47a8-884a-8ddc05f375ed-serviceca\") pod \"node-ca-m4n88\" (UID: \"afe27658-50c9-47a8-884a-8ddc05f375ed\") " pod="openshift-image-registry/node-ca-m4n88"
Sep 08 16:30:18.416703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415561    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10038ceb-d586-4fb2-99e2-dc326d9daa8c-system-cni-dir\") pod \"multus-additional-cni-plugins-62zlz\" (UID: \"10038ceb-d586-4fb2-99e2-dc326d9daa8c\") " pod="openshift-multus/multus-additional-cni-plugins-62zlz"
Sep 08 16:30:18.416703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415588    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-var-lib-kubelet\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.416703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415602    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/10038ceb-d586-4fb2-99e2-dc326d9daa8c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-62zlz\" (UID: \"10038ceb-d586-4fb2-99e2-dc326d9daa8c\") " pod="openshift-multus/multus-additional-cni-plugins-62zlz"
Sep 08 16:30:18.416703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415626    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ad2a7c50-5f92-449d-a650-ed26da4359b7-hosts-file\") pod \"node-resolver-rj64h\" (UID: \"ad2a7c50-5f92-449d-a650-ed26da4359b7\") " pod="openshift-dns/node-resolver-rj64h"
Sep 08 16:30:18.416703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415626    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-multus-cni-dir\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.416703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415478    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-run\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.416703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415689    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d1f5282-a130-4c07-86b0-c70f99879223-sys-fs\") pod \"aws-ebs-csi-driver-node-nrmb6\" (UID: \"3d1f5282-a130-4c07-86b0-c70f99879223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6"
Sep 08 16:30:18.416703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415695    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-system-cni-dir\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.416703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415726    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d1f5282-a130-4c07-86b0-c70f99879223-etc-selinux\") pod \"aws-ebs-csi-driver-node-nrmb6\" (UID: \"3d1f5282-a130-4c07-86b0-c70f99879223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6"
Sep 08 16:30:18.416703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415733    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-host-var-lib-kubelet\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.417471 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415742    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-host\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.417471 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415843    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-etc-kubernetes\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.417471 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.415975    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10038ceb-d586-4fb2-99e2-dc326d9daa8c-system-cni-dir\") pod \"multus-additional-cni-plugins-62zlz\" (UID: \"10038ceb-d586-4fb2-99e2-dc326d9daa8c\") " pod="openshift-multus/multus-additional-cni-plugins-62zlz"
Sep 08 16:30:18.417471 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416054    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ad2a7c50-5f92-449d-a650-ed26da4359b7-hosts-file\") pod \"node-resolver-rj64h\" (UID: \"ad2a7c50-5f92-449d-a650-ed26da4359b7\") " pod="openshift-dns/node-resolver-rj64h"
Sep 08 16:30:18.417471 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416067    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-var-lib-kubelet\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.417471 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416129    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-cnibin\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.417471 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416157    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-host-run-multus-certs\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.417471 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416184    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-etc-systemd\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.417471 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416210    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-host-run-k8s-cni-cncf-io\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.417471 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416218    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-cnibin\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.417471 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416236    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0abcdd93-6ae1-4e51-a19b-911c06152822-kubelet-config\") pod \"global-pull-secret-syncer-g67mm\" (UID: \"0abcdd93-6ae1-4e51-a19b-911c06152822\") " pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:18.417471 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416265    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d1f5282-a130-4c07-86b0-c70f99879223-registration-dir\") pod \"aws-ebs-csi-driver-node-nrmb6\" (UID: \"3d1f5282-a130-4c07-86b0-c70f99879223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6"
Sep 08 16:30:18.417471 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416272    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-etc-systemd\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.417471 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416288    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff68e8af-fa2a-4b97-a923-3db24f91d137-cni-binary-copy\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.417471 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416294    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xf79\" (UniqueName: \"kubernetes.io/projected/d5c4bf16-72bf-41a9-bc8a-58cc50aa0cba-kube-api-access-5xf79\") pod \"iptables-alerter-r4kbv\" (UID: \"d5c4bf16-72bf-41a9-bc8a-58cc50aa0cba\") " pod="openshift-network-operator/iptables-alerter-r4kbv"
Sep 08 16:30:18.417471 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416316    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-host-run-multus-certs\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.417471 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416322    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10038ceb-d586-4fb2-99e2-dc326d9daa8c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-62zlz\" (UID: \"10038ceb-d586-4fb2-99e2-dc326d9daa8c\") " pod="openshift-multus/multus-additional-cni-plugins-62zlz"
Sep 08 16:30:18.417471 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416357    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0abcdd93-6ae1-4e51-a19b-911c06152822-kubelet-config\") pod \"global-pull-secret-syncer-g67mm\" (UID: \"0abcdd93-6ae1-4e51-a19b-911c06152822\") " pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:18.418034 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416319    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/afe27658-50c9-47a8-884a-8ddc05f375ed-serviceca\") pod \"node-ca-m4n88\" (UID: \"afe27658-50c9-47a8-884a-8ddc05f375ed\") " pod="openshift-image-registry/node-ca-m4n88"
Sep 08 16:30:18.418034 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416357    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-host-run-k8s-cni-cncf-io\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.418034 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416359    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/10038ceb-d586-4fb2-99e2-dc326d9daa8c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-62zlz\" (UID: \"10038ceb-d586-4fb2-99e2-dc326d9daa8c\") " pod="openshift-multus/multus-additional-cni-plugins-62zlz"
Sep 08 16:30:18.418034 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416398    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d1f5282-a130-4c07-86b0-c70f99879223-registration-dir\") pod \"aws-ebs-csi-driver-node-nrmb6\" (UID: \"3d1f5282-a130-4c07-86b0-c70f99879223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6"
Sep 08 16:30:18.418034 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416430    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-sys\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.418034 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416461    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-host-var-lib-cni-bin\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.418034 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416486    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-host-var-lib-cni-multus\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.418034 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416490    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10038ceb-d586-4fb2-99e2-dc326d9daa8c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-62zlz\" (UID: \"10038ceb-d586-4fb2-99e2-dc326d9daa8c\") " pod="openshift-multus/multus-additional-cni-plugins-62zlz"
Sep 08 16:30:18.418034 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416528    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-host-var-lib-cni-multus\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.418034 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416548    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-sys\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.418034 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416557    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d5c4bf16-72bf-41a9-bc8a-58cc50aa0cba-iptables-alerter-script\") pod \"iptables-alerter-r4kbv\" (UID: \"d5c4bf16-72bf-41a9-bc8a-58cc50aa0cba\") " pod="openshift-network-operator/iptables-alerter-r4kbv"
Sep 08 16:30:18.418034 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416587    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-host-var-lib-cni-bin\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.418034 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416591    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-lib-modules\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.418034 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416620    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad2a7c50-5f92-449d-a650-ed26da4359b7-tmp-dir\") pod \"node-resolver-rj64h\" (UID: \"ad2a7c50-5f92-449d-a650-ed26da4359b7\") " pod="openshift-dns/node-resolver-rj64h"
Sep 08 16:30:18.418034 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416649    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-host-run-netns\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.418034 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416675    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ff68e8af-fa2a-4b97-a923-3db24f91d137-multus-daemon-config\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.418034 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416679    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10038ceb-d586-4fb2-99e2-dc326d9daa8c-cni-binary-copy\") pod \"multus-additional-cni-plugins-62zlz\" (UID: \"10038ceb-d586-4fb2-99e2-dc326d9daa8c\") " pod="openshift-multus/multus-additional-cni-plugins-62zlz"
Sep 08 16:30:18.418557 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416691    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b925da05-aa52-48c7-8dc0-839d562e1d01-lib-modules\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.418557 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416699    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10038ceb-d586-4fb2-99e2-dc326d9daa8c-cnibin\") pod \"multus-additional-cni-plugins-62zlz\" (UID: \"10038ceb-d586-4fb2-99e2-dc326d9daa8c\") " pod="openshift-multus/multus-additional-cni-plugins-62zlz"
Sep 08 16:30:18.418557 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416735    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-host-run-netns\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.418557 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416739    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-etc-kubernetes\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.418557 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416744    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10038ceb-d586-4fb2-99e2-dc326d9daa8c-cnibin\") pod \"multus-additional-cni-plugins-62zlz\" (UID: \"10038ceb-d586-4fb2-99e2-dc326d9daa8c\") " pod="openshift-multus/multus-additional-cni-plugins-62zlz"
Sep 08 16:30:18.418557 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416767    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d1f5282-a130-4c07-86b0-c70f99879223-socket-dir\") pod \"aws-ebs-csi-driver-node-nrmb6\" (UID: \"3d1f5282-a130-4c07-86b0-c70f99879223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6"
Sep 08 16:30:18.418557 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416792    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0abcdd93-6ae1-4e51-a19b-911c06152822-dbus\") pod \"global-pull-secret-syncer-g67mm\" (UID: \"0abcdd93-6ae1-4e51-a19b-911c06152822\") " pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:18.418557 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416818    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0abcdd93-6ae1-4e51-a19b-911c06152822-original-pull-secret\") pod \"global-pull-secret-syncer-g67mm\" (UID: \"0abcdd93-6ae1-4e51-a19b-911c06152822\") " pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:18.418557 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:18.416944    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:30:18.418557 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.416984    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/10038ceb-d586-4fb2-99e2-dc326d9daa8c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-62zlz\" (UID: \"10038ceb-d586-4fb2-99e2-dc326d9daa8c\") " pod="openshift-multus/multus-additional-cni-plugins-62zlz"
Sep 08 16:30:18.418557 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:18.417004    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0abcdd93-6ae1-4e51-a19b-911c06152822-original-pull-secret podName:0abcdd93-6ae1-4e51-a19b-911c06152822 nodeName:}" failed. No retries permitted until 2026-09-08 16:30:18.916983099 +0000 UTC m=+3.079954533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0abcdd93-6ae1-4e51-a19b-911c06152822-original-pull-secret") pod "global-pull-secret-syncer-g67mm" (UID: "0abcdd93-6ae1-4e51-a19b-911c06152822") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:30:18.418557 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.417053    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff68e8af-fa2a-4b97-a923-3db24f91d137-etc-kubernetes\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.418557 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.417083    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad2a7c50-5f92-449d-a650-ed26da4359b7-tmp-dir\") pod \"node-resolver-rj64h\" (UID: \"ad2a7c50-5f92-449d-a650-ed26da4359b7\") " pod="openshift-dns/node-resolver-rj64h"
Sep 08 16:30:18.418557 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.417112    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0abcdd93-6ae1-4e51-a19b-911c06152822-dbus\") pod \"global-pull-secret-syncer-g67mm\" (UID: \"0abcdd93-6ae1-4e51-a19b-911c06152822\") " pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:18.418557 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.417136    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d5c4bf16-72bf-41a9-bc8a-58cc50aa0cba-iptables-alerter-script\") pod \"iptables-alerter-r4kbv\" (UID: \"d5c4bf16-72bf-41a9-bc8a-58cc50aa0cba\") " pod="openshift-network-operator/iptables-alerter-r4kbv"
Sep 08 16:30:18.418557 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.417113    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d1f5282-a130-4c07-86b0-c70f99879223-socket-dir\") pod \"aws-ebs-csi-driver-node-nrmb6\" (UID: \"3d1f5282-a130-4c07-86b0-c70f99879223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6"
Sep 08 16:30:18.418557 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.417227    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ff68e8af-fa2a-4b97-a923-3db24f91d137-multus-daemon-config\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.419074 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.417898    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b925da05-aa52-48c7-8dc0-839d562e1d01-tmp\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.419074 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.417966    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b925da05-aa52-48c7-8dc0-839d562e1d01-etc-tuned\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.424757 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:18.424663    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:30:18.424757 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:18.424689    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:30:18.424757 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:18.424704    2557 projected.go:196] Error preparing data for projected volume kube-api-access-hvcz8 for pod openshift-network-diagnostics/network-check-target-mz7tj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:30:18.425033 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:18.424764    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50d7b024-55d7-4077-8584-9657bf3008b5-kube-api-access-hvcz8 podName:50d7b024-55d7-4077-8584-9657bf3008b5 nodeName:}" failed. No retries permitted until 2026-09-08 16:30:18.924746849 +0000 UTC m=+3.087718291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hvcz8" (UniqueName: "kubernetes.io/projected/50d7b024-55d7-4077-8584-9657bf3008b5-kube-api-access-hvcz8") pod "network-check-target-mz7tj" (UID: "50d7b024-55d7-4077-8584-9657bf3008b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:30:18.425033 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.424955    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt2vb\" (UniqueName: \"kubernetes.io/projected/b925da05-aa52-48c7-8dc0-839d562e1d01-kube-api-access-bt2vb\") pod \"tuned-dxt9v\" (UID: \"b925da05-aa52-48c7-8dc0-839d562e1d01\") " pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.425476 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.425452    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhbqb\" (UniqueName: \"kubernetes.io/projected/3d1f5282-a130-4c07-86b0-c70f99879223-kube-api-access-bhbqb\") pod \"aws-ebs-csi-driver-node-nrmb6\" (UID: \"3d1f5282-a130-4c07-86b0-c70f99879223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6"
Sep 08 16:30:18.426566 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.426540    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xf79\" (UniqueName: \"kubernetes.io/projected/d5c4bf16-72bf-41a9-bc8a-58cc50aa0cba-kube-api-access-5xf79\") pod \"iptables-alerter-r4kbv\" (UID: \"d5c4bf16-72bf-41a9-bc8a-58cc50aa0cba\") " pod="openshift-network-operator/iptables-alerter-r4kbv"
Sep 08 16:30:18.426887 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.426872    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph7sw\" (UniqueName: \"kubernetes.io/projected/ad2a7c50-5f92-449d-a650-ed26da4359b7-kube-api-access-ph7sw\") pod \"node-resolver-rj64h\" (UID: \"ad2a7c50-5f92-449d-a650-ed26da4359b7\") " pod="openshift-dns/node-resolver-rj64h"
Sep 08 16:30:18.427095 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.427079    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g756r\" (UniqueName: \"kubernetes.io/projected/ff68e8af-fa2a-4b97-a923-3db24f91d137-kube-api-access-g756r\") pod \"multus-q8kt4\" (UID: \"ff68e8af-fa2a-4b97-a923-3db24f91d137\") " pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.427423 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.427404    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2whdh\" (UniqueName: \"kubernetes.io/projected/afe27658-50c9-47a8-884a-8ddc05f375ed-kube-api-access-2whdh\") pod \"node-ca-m4n88\" (UID: \"afe27658-50c9-47a8-884a-8ddc05f375ed\") " pod="openshift-image-registry/node-ca-m4n88"
Sep 08 16:30:18.427648 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.427629    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6xvx\" (UniqueName: \"kubernetes.io/projected/10038ceb-d586-4fb2-99e2-dc326d9daa8c-kube-api-access-k6xvx\") pod \"multus-additional-cni-plugins-62zlz\" (UID: \"10038ceb-d586-4fb2-99e2-dc326d9daa8c\") " pod="openshift-multus/multus-additional-cni-plugins-62zlz"
Sep 08 16:30:18.508072 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.507977    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:18.514347 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.514313    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pqhsk"
Sep 08 16:30:18.518414 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:18.518388    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e9b515_3988_4240_a220_e3325ecfbcee.slice/crio-81bcc0d3b2b719a075ca8cafb1abed6c91fbace9a9390770b4121fb3df8e3ed0 WatchSource:0}: Error finding container 81bcc0d3b2b719a075ca8cafb1abed6c91fbace9a9390770b4121fb3df8e3ed0: Status 404 returned error can't find the container with id 81bcc0d3b2b719a075ca8cafb1abed6c91fbace9a9390770b4121fb3df8e3ed0
Sep 08 16:30:18.523512 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.523288    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6"
Sep 08 16:30:18.527493 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.527473    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m4n88"
Sep 08 16:30:18.533924 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.533699    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-62zlz"
Sep 08 16:30:18.536877 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:18.536843    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1f5282_a130_4c07_86b0_c70f99879223.slice/crio-752dccafb36ffde60659919f996214b4949de3da565e726af3937dad814a5319 WatchSource:0}: Error finding container 752dccafb36ffde60659919f996214b4949de3da565e726af3937dad814a5319: Status 404 returned error can't find the container with id 752dccafb36ffde60659919f996214b4949de3da565e726af3937dad814a5319
Sep 08 16:30:18.537884 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:18.537770    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe27658_50c9_47a8_884a_8ddc05f375ed.slice/crio-e82dc8382a82683982ee37ceef07a54f4e7039424df31722d00a88831283a51c WatchSource:0}: Error finding container e82dc8382a82683982ee37ceef07a54f4e7039424df31722d00a88831283a51c: Status 404 returned error can't find the container with id e82dc8382a82683982ee37ceef07a54f4e7039424df31722d00a88831283a51c
Sep 08 16:30:18.539746 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.539727    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dxt9v"
Sep 08 16:30:18.544979 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:18.544893    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10038ceb_d586_4fb2_99e2_dc326d9daa8c.slice/crio-a9a3bba532b0554bc9bb1eaa41b80fa198f71d28fbe34ee5b3ecd90c61828643 WatchSource:0}: Error finding container a9a3bba532b0554bc9bb1eaa41b80fa198f71d28fbe34ee5b3ecd90c61828643: Status 404 returned error can't find the container with id a9a3bba532b0554bc9bb1eaa41b80fa198f71d28fbe34ee5b3ecd90c61828643
Sep 08 16:30:18.546295 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.546275    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r4kbv"
Sep 08 16:30:18.549704 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:18.549679    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb925da05_aa52_48c7_8dc0_839d562e1d01.slice/crio-db074503316055de4db46fd1b9c06bad9be692e5817934a4f1dbca3908808a87 WatchSource:0}: Error finding container db074503316055de4db46fd1b9c06bad9be692e5817934a4f1dbca3908808a87: Status 404 returned error can't find the container with id db074503316055de4db46fd1b9c06bad9be692e5817934a4f1dbca3908808a87
Sep 08 16:30:18.554197 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.554178    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rj64h"
Sep 08 16:30:18.557710 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.557684    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q8kt4"
Sep 08 16:30:18.558420 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:18.558223    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c4bf16_72bf_41a9_bc8a_58cc50aa0cba.slice/crio-cba30e8a82dd20a08ff7fa7b2e2f153bf121957768c4f5b77badcba3ffc07300 WatchSource:0}: Error finding container cba30e8a82dd20a08ff7fa7b2e2f153bf121957768c4f5b77badcba3ffc07300: Status 404 returned error can't find the container with id cba30e8a82dd20a08ff7fa7b2e2f153bf121957768c4f5b77badcba3ffc07300
Sep 08 16:30:18.565551 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:18.565527    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2a7c50_5f92_449d_a650_ed26da4359b7.slice/crio-2ac7c991af3f7582f6155de07d408f412bdb4caa164dfdf4ce75a4e110f5a884 WatchSource:0}: Error finding container 2ac7c991af3f7582f6155de07d408f412bdb4caa164dfdf4ce75a4e110f5a884: Status 404 returned error can't find the container with id 2ac7c991af3f7582f6155de07d408f412bdb4caa164dfdf4ce75a4e110f5a884
Sep 08 16:30:18.568174 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:18.568151    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff68e8af_fa2a_4b97_a923_3db24f91d137.slice/crio-8f473dd18a3b77a88d142973f26acacadbda348209253f2fc437450049ffa9fa WatchSource:0}: Error finding container 8f473dd18a3b77a88d142973f26acacadbda348209253f2fc437450049ffa9fa: Status 404 returned error can't find the container with id 8f473dd18a3b77a88d142973f26acacadbda348209253f2fc437450049ffa9fa
Sep 08 16:30:18.819020 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.818926    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48a3c156-d793-4586-9ef9-c6770eb982d0-metrics-certs\") pod \"network-metrics-daemon-fm6rg\" (UID: \"48a3c156-d793-4586-9ef9-c6770eb982d0\") " pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:18.819167 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:18.819092    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:30:18.819167 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:18.819163    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48a3c156-d793-4586-9ef9-c6770eb982d0-metrics-certs podName:48a3c156-d793-4586-9ef9-c6770eb982d0 nodeName:}" failed. No retries permitted until 2026-09-08 16:30:19.819142778 +0000 UTC m=+3.982114224 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48a3c156-d793-4586-9ef9-c6770eb982d0-metrics-certs") pod "network-metrics-daemon-fm6rg" (UID: "48a3c156-d793-4586-9ef9-c6770eb982d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:30:18.919354 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:18.919313    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0abcdd93-6ae1-4e51-a19b-911c06152822-original-pull-secret\") pod \"global-pull-secret-syncer-g67mm\" (UID: \"0abcdd93-6ae1-4e51-a19b-911c06152822\") " pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:18.919548 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:18.919516    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:30:18.919614 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:18.919603    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0abcdd93-6ae1-4e51-a19b-911c06152822-original-pull-secret podName:0abcdd93-6ae1-4e51-a19b-911c06152822 nodeName:}" failed. No retries permitted until 2026-09-08 16:30:19.91958235 +0000 UTC m=+4.082553785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0abcdd93-6ae1-4e51-a19b-911c06152822-original-pull-secret") pod "global-pull-secret-syncer-g67mm" (UID: "0abcdd93-6ae1-4e51-a19b-911c06152822") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:30:19.019765 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:19.019725    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvcz8\" (UniqueName: \"kubernetes.io/projected/50d7b024-55d7-4077-8584-9657bf3008b5-kube-api-access-hvcz8\") pod \"network-check-target-mz7tj\" (UID: \"50d7b024-55d7-4077-8584-9657bf3008b5\") " pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:19.019970 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:19.019918    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:30:19.019970 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:19.019943    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:30:19.019970 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:19.019956    2557 projected.go:196] Error preparing data for projected volume kube-api-access-hvcz8 for pod openshift-network-diagnostics/network-check-target-mz7tj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:30:19.020093 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:19.020012    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50d7b024-55d7-4077-8584-9657bf3008b5-kube-api-access-hvcz8 podName:50d7b024-55d7-4077-8584-9657bf3008b5 nodeName:}" failed. No retries permitted until 2026-09-08 16:30:20.019994254 +0000 UTC m=+4.182965696 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvcz8" (UniqueName: "kubernetes.io/projected/50d7b024-55d7-4077-8584-9657bf3008b5-kube-api-access-hvcz8") pod "network-check-target-mz7tj" (UID: "50d7b024-55d7-4077-8584-9657bf3008b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:30:19.268886 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:19.268824    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 16:25:18 +0000 UTC" deadline="2028-03-19 00:59:52.529542931 +0000 UTC"
Sep 08 16:30:19.268886 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:19.268880    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13376h29m33.260668383s"
Sep 08 16:30:19.340410 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:19.340103    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:19.340410 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:19.340244    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz7tj" podUID="50d7b024-55d7-4077-8584-9657bf3008b5"
Sep 08 16:30:19.340410 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:19.340329    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:19.340410 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:19.340403    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g67mm" podUID="0abcdd93-6ae1-4e51-a19b-911c06152822"
Sep 08 16:30:19.357140 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:19.357013    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd" event={"ID":"41e9b515-3988-4240-a220-e3325ecfbcee","Type":"ContainerStarted","Data":"81bcc0d3b2b719a075ca8cafb1abed6c91fbace9a9390770b4121fb3df8e3ed0"}
Sep 08 16:30:19.363185 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:19.363148    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q8kt4" event={"ID":"ff68e8af-fa2a-4b97-a923-3db24f91d137","Type":"ContainerStarted","Data":"8f473dd18a3b77a88d142973f26acacadbda348209253f2fc437450049ffa9fa"}
Sep 08 16:30:19.366379 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:19.366347    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-62zlz" event={"ID":"10038ceb-d586-4fb2-99e2-dc326d9daa8c","Type":"ContainerStarted","Data":"a9a3bba532b0554bc9bb1eaa41b80fa198f71d28fbe34ee5b3ecd90c61828643"}
Sep 08 16:30:19.373296 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:19.373264    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m4n88" event={"ID":"afe27658-50c9-47a8-884a-8ddc05f375ed","Type":"ContainerStarted","Data":"e82dc8382a82683982ee37ceef07a54f4e7039424df31722d00a88831283a51c"}
Sep 08 16:30:19.375746 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:19.375715    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pqhsk" event={"ID":"bd8c514b-7ca3-43e4-b87c-8a811c04efe8","Type":"ContainerStarted","Data":"5d3145d16de83cf321180c2eb3ab884d2b71b580c0af6f6784bbb45e9b5c808b"}
Sep 08 16:30:19.380246 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:19.380216    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rj64h" event={"ID":"ad2a7c50-5f92-449d-a650-ed26da4359b7","Type":"ContainerStarted","Data":"2ac7c991af3f7582f6155de07d408f412bdb4caa164dfdf4ce75a4e110f5a884"}
Sep 08 16:30:19.381845 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:19.381812    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r4kbv" event={"ID":"d5c4bf16-72bf-41a9-bc8a-58cc50aa0cba","Type":"ContainerStarted","Data":"cba30e8a82dd20a08ff7fa7b2e2f153bf121957768c4f5b77badcba3ffc07300"}
Sep 08 16:30:19.391384 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:19.391356    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dxt9v" event={"ID":"b925da05-aa52-48c7-8dc0-839d562e1d01","Type":"ContainerStarted","Data":"db074503316055de4db46fd1b9c06bad9be692e5817934a4f1dbca3908808a87"}
Sep 08 16:30:19.396284 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:19.396255    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6" event={"ID":"3d1f5282-a130-4c07-86b0-c70f99879223","Type":"ContainerStarted","Data":"752dccafb36ffde60659919f996214b4949de3da565e726af3937dad814a5319"}
Sep 08 16:30:19.828082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:19.828033    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48a3c156-d793-4586-9ef9-c6770eb982d0-metrics-certs\") pod \"network-metrics-daemon-fm6rg\" (UID: \"48a3c156-d793-4586-9ef9-c6770eb982d0\") " pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:19.828275 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:19.828242    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:30:19.828344 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:19.828324    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48a3c156-d793-4586-9ef9-c6770eb982d0-metrics-certs podName:48a3c156-d793-4586-9ef9-c6770eb982d0 nodeName:}" failed. No retries permitted until 2026-09-08 16:30:21.828303422 +0000 UTC m=+5.991274871 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48a3c156-d793-4586-9ef9-c6770eb982d0-metrics-certs") pod "network-metrics-daemon-fm6rg" (UID: "48a3c156-d793-4586-9ef9-c6770eb982d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:30:19.929941 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:19.929322    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0abcdd93-6ae1-4e51-a19b-911c06152822-original-pull-secret\") pod \"global-pull-secret-syncer-g67mm\" (UID: \"0abcdd93-6ae1-4e51-a19b-911c06152822\") " pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:19.929941 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:19.929482    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:30:19.929941 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:19.929551    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0abcdd93-6ae1-4e51-a19b-911c06152822-original-pull-secret podName:0abcdd93-6ae1-4e51-a19b-911c06152822 nodeName:}" failed. No retries permitted until 2026-09-08 16:30:21.929530889 +0000 UTC m=+6.092502349 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0abcdd93-6ae1-4e51-a19b-911c06152822-original-pull-secret") pod "global-pull-secret-syncer-g67mm" (UID: "0abcdd93-6ae1-4e51-a19b-911c06152822") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:30:20.030558 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:20.030456    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvcz8\" (UniqueName: \"kubernetes.io/projected/50d7b024-55d7-4077-8584-9657bf3008b5-kube-api-access-hvcz8\") pod \"network-check-target-mz7tj\" (UID: \"50d7b024-55d7-4077-8584-9657bf3008b5\") " pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:20.030724 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:20.030672    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:30:20.030724 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:20.030694    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:30:20.030724 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:20.030708    2557 projected.go:196] Error preparing data for projected volume kube-api-access-hvcz8 for pod openshift-network-diagnostics/network-check-target-mz7tj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:30:20.030932 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:20.030774    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50d7b024-55d7-4077-8584-9657bf3008b5-kube-api-access-hvcz8 podName:50d7b024-55d7-4077-8584-9657bf3008b5 nodeName:}" failed. No retries permitted until 2026-09-08 16:30:22.03075219 +0000 UTC m=+6.193723634 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvcz8" (UniqueName: "kubernetes.io/projected/50d7b024-55d7-4077-8584-9657bf3008b5-kube-api-access-hvcz8") pod "network-check-target-mz7tj" (UID: "50d7b024-55d7-4077-8584-9657bf3008b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:30:20.269592 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:20.269511    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 16:25:18 +0000 UTC" deadline="2028-04-12 17:02:06.847793855 +0000 UTC"
Sep 08 16:30:20.269592 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:20.269590    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13968h31m46.57820983s"
Sep 08 16:30:20.341227 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:20.341141    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:20.341386 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:20.341342    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fm6rg" podUID="48a3c156-d793-4586-9ef9-c6770eb982d0"
Sep 08 16:30:21.339946 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:21.339904    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:21.340457 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:21.340040    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz7tj" podUID="50d7b024-55d7-4077-8584-9657bf3008b5"
Sep 08 16:30:21.340534 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:21.340457    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:21.340594 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:21.340545    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g67mm" podUID="0abcdd93-6ae1-4e51-a19b-911c06152822"
Sep 08 16:30:21.846376 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:21.846335    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48a3c156-d793-4586-9ef9-c6770eb982d0-metrics-certs\") pod \"network-metrics-daemon-fm6rg\" (UID: \"48a3c156-d793-4586-9ef9-c6770eb982d0\") " pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:21.846558 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:21.846528    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:30:21.846628 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:21.846597    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48a3c156-d793-4586-9ef9-c6770eb982d0-metrics-certs podName:48a3c156-d793-4586-9ef9-c6770eb982d0 nodeName:}" failed. No retries permitted until 2026-09-08 16:30:25.846579808 +0000 UTC m=+10.009551236 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48a3c156-d793-4586-9ef9-c6770eb982d0-metrics-certs") pod "network-metrics-daemon-fm6rg" (UID: "48a3c156-d793-4586-9ef9-c6770eb982d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:30:21.947291 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:21.947246    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0abcdd93-6ae1-4e51-a19b-911c06152822-original-pull-secret\") pod \"global-pull-secret-syncer-g67mm\" (UID: \"0abcdd93-6ae1-4e51-a19b-911c06152822\") " pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:21.947474 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:21.947432    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:30:21.947537 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:21.947505    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0abcdd93-6ae1-4e51-a19b-911c06152822-original-pull-secret podName:0abcdd93-6ae1-4e51-a19b-911c06152822 nodeName:}" failed. No retries permitted until 2026-09-08 16:30:25.947489018 +0000 UTC m=+10.110460445 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0abcdd93-6ae1-4e51-a19b-911c06152822-original-pull-secret") pod "global-pull-secret-syncer-g67mm" (UID: "0abcdd93-6ae1-4e51-a19b-911c06152822") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:30:22.047967 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:22.047929    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvcz8\" (UniqueName: \"kubernetes.io/projected/50d7b024-55d7-4077-8584-9657bf3008b5-kube-api-access-hvcz8\") pod \"network-check-target-mz7tj\" (UID: \"50d7b024-55d7-4077-8584-9657bf3008b5\") " pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:22.048151 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:22.048127    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:30:22.048208 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:22.048157    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:30:22.048208 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:22.048169    2557 projected.go:196] Error preparing data for projected volume kube-api-access-hvcz8 for pod openshift-network-diagnostics/network-check-target-mz7tj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:30:22.048283 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:22.048223    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50d7b024-55d7-4077-8584-9657bf3008b5-kube-api-access-hvcz8 podName:50d7b024-55d7-4077-8584-9657bf3008b5 nodeName:}" failed. No retries permitted until 2026-09-08 16:30:26.048207029 +0000 UTC m=+10.211178458 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvcz8" (UniqueName: "kubernetes.io/projected/50d7b024-55d7-4077-8584-9657bf3008b5-kube-api-access-hvcz8") pod "network-check-target-mz7tj" (UID: "50d7b024-55d7-4077-8584-9657bf3008b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:30:22.339945 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:22.339913    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:22.340131 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:22.340061    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fm6rg" podUID="48a3c156-d793-4586-9ef9-c6770eb982d0"
Sep 08 16:30:23.340043 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:23.339923    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:23.340239 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:23.340147    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz7tj" podUID="50d7b024-55d7-4077-8584-9657bf3008b5"
Sep 08 16:30:23.340239 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:23.340196    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:23.340645 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:23.340265    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g67mm" podUID="0abcdd93-6ae1-4e51-a19b-911c06152822"
Sep 08 16:30:24.340073 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:24.339976    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:24.340286 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:24.340128    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fm6rg" podUID="48a3c156-d793-4586-9ef9-c6770eb982d0"
Sep 08 16:30:25.339538 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:25.339499    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:25.339538 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:25.339536    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:25.339766 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:25.339640    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g67mm" podUID="0abcdd93-6ae1-4e51-a19b-911c06152822"
Sep 08 16:30:25.339766 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:25.339721    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz7tj" podUID="50d7b024-55d7-4077-8584-9657bf3008b5"
Sep 08 16:30:25.882187 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:25.881506    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48a3c156-d793-4586-9ef9-c6770eb982d0-metrics-certs\") pod \"network-metrics-daemon-fm6rg\" (UID: \"48a3c156-d793-4586-9ef9-c6770eb982d0\") " pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:25.882187 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:25.881714    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:30:25.882187 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:25.881784    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48a3c156-d793-4586-9ef9-c6770eb982d0-metrics-certs podName:48a3c156-d793-4586-9ef9-c6770eb982d0 nodeName:}" failed. No retries permitted until 2026-09-08 16:30:33.881764944 +0000 UTC m=+18.044736374 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48a3c156-d793-4586-9ef9-c6770eb982d0-metrics-certs") pod "network-metrics-daemon-fm6rg" (UID: "48a3c156-d793-4586-9ef9-c6770eb982d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:30:25.982637 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:25.982533    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0abcdd93-6ae1-4e51-a19b-911c06152822-original-pull-secret\") pod \"global-pull-secret-syncer-g67mm\" (UID: \"0abcdd93-6ae1-4e51-a19b-911c06152822\") " pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:25.982792 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:25.982708    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:30:25.982792 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:25.982778    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0abcdd93-6ae1-4e51-a19b-911c06152822-original-pull-secret podName:0abcdd93-6ae1-4e51-a19b-911c06152822 nodeName:}" failed. No retries permitted until 2026-09-08 16:30:33.982758853 +0000 UTC m=+18.145730285 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0abcdd93-6ae1-4e51-a19b-911c06152822-original-pull-secret") pod "global-pull-secret-syncer-g67mm" (UID: "0abcdd93-6ae1-4e51-a19b-911c06152822") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:30:26.083581 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:26.083543    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvcz8\" (UniqueName: \"kubernetes.io/projected/50d7b024-55d7-4077-8584-9657bf3008b5-kube-api-access-hvcz8\") pod \"network-check-target-mz7tj\" (UID: \"50d7b024-55d7-4077-8584-9657bf3008b5\") " pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:26.083722 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:26.083709    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:30:26.083768 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:26.083731    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:30:26.083768 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:26.083745    2557 projected.go:196] Error preparing data for projected volume kube-api-access-hvcz8 for pod openshift-network-diagnostics/network-check-target-mz7tj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:30:26.083903 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:26.083809    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50d7b024-55d7-4077-8584-9657bf3008b5-kube-api-access-hvcz8 podName:50d7b024-55d7-4077-8584-9657bf3008b5 nodeName:}" failed. No retries permitted until 2026-09-08 16:30:34.083791228 +0000 UTC m=+18.246762658 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvcz8" (UniqueName: "kubernetes.io/projected/50d7b024-55d7-4077-8584-9657bf3008b5-kube-api-access-hvcz8") pod "network-check-target-mz7tj" (UID: "50d7b024-55d7-4077-8584-9657bf3008b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:30:26.341351 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:26.340948    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:26.341351 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:26.341046    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fm6rg" podUID="48a3c156-d793-4586-9ef9-c6770eb982d0"
Sep 08 16:30:27.340295 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:27.340258    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:27.340751 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:27.340366    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz7tj" podUID="50d7b024-55d7-4077-8584-9657bf3008b5"
Sep 08 16:30:27.340751 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:27.340702    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:27.340853 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:27.340797    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g67mm" podUID="0abcdd93-6ae1-4e51-a19b-911c06152822"
Sep 08 16:30:28.342420 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:28.342379    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:28.342845 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:28.342529    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fm6rg" podUID="48a3c156-d793-4586-9ef9-c6770eb982d0"
Sep 08 16:30:29.339799 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:29.339762    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:29.339977 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:29.339898    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz7tj" podUID="50d7b024-55d7-4077-8584-9657bf3008b5"
Sep 08 16:30:29.340019 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:29.339981    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:29.340050 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:29.340033    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g67mm" podUID="0abcdd93-6ae1-4e51-a19b-911c06152822"
Sep 08 16:30:30.339681 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:30.339644    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:30.340123 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:30.339786    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fm6rg" podUID="48a3c156-d793-4586-9ef9-c6770eb982d0"
Sep 08 16:30:31.339953 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:31.339912    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:31.340314 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:31.339912    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:31.340314 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:31.340033    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g67mm" podUID="0abcdd93-6ae1-4e51-a19b-911c06152822"
Sep 08 16:30:31.340314 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:31.340129    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz7tj" podUID="50d7b024-55d7-4077-8584-9657bf3008b5"
Sep 08 16:30:32.339971 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:32.339939    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:32.340431 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:32.340084    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fm6rg" podUID="48a3c156-d793-4586-9ef9-c6770eb982d0"
Sep 08 16:30:33.339793 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:33.339750    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:33.340051 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:33.339756    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:33.340051 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:33.339897    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz7tj" podUID="50d7b024-55d7-4077-8584-9657bf3008b5"
Sep 08 16:30:33.340051 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:33.340003    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g67mm" podUID="0abcdd93-6ae1-4e51-a19b-911c06152822"
Sep 08 16:30:33.937560 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:33.937522    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48a3c156-d793-4586-9ef9-c6770eb982d0-metrics-certs\") pod \"network-metrics-daemon-fm6rg\" (UID: \"48a3c156-d793-4586-9ef9-c6770eb982d0\") " pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:33.937730 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:33.937699    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:30:33.937799 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:33.937765    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48a3c156-d793-4586-9ef9-c6770eb982d0-metrics-certs podName:48a3c156-d793-4586-9ef9-c6770eb982d0 nodeName:}" failed. No retries permitted until 2026-09-08 16:30:49.937750089 +0000 UTC m=+34.100721518 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48a3c156-d793-4586-9ef9-c6770eb982d0-metrics-certs") pod "network-metrics-daemon-fm6rg" (UID: "48a3c156-d793-4586-9ef9-c6770eb982d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:30:34.038581 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:34.038537    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0abcdd93-6ae1-4e51-a19b-911c06152822-original-pull-secret\") pod \"global-pull-secret-syncer-g67mm\" (UID: \"0abcdd93-6ae1-4e51-a19b-911c06152822\") " pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:34.038761 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:34.038715    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:30:34.038815 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:34.038795    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0abcdd93-6ae1-4e51-a19b-911c06152822-original-pull-secret podName:0abcdd93-6ae1-4e51-a19b-911c06152822 nodeName:}" failed. No retries permitted until 2026-09-08 16:30:50.038773844 +0000 UTC m=+34.201745288 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0abcdd93-6ae1-4e51-a19b-911c06152822-original-pull-secret") pod "global-pull-secret-syncer-g67mm" (UID: "0abcdd93-6ae1-4e51-a19b-911c06152822") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:30:34.139594 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:34.139562    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvcz8\" (UniqueName: \"kubernetes.io/projected/50d7b024-55d7-4077-8584-9657bf3008b5-kube-api-access-hvcz8\") pod \"network-check-target-mz7tj\" (UID: \"50d7b024-55d7-4077-8584-9657bf3008b5\") " pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:34.139763 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:34.139723    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:30:34.139763 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:34.139744    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:30:34.139763 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:34.139759    2557 projected.go:196] Error preparing data for projected volume kube-api-access-hvcz8 for pod openshift-network-diagnostics/network-check-target-mz7tj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:30:34.139906 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:34.139805    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50d7b024-55d7-4077-8584-9657bf3008b5-kube-api-access-hvcz8 podName:50d7b024-55d7-4077-8584-9657bf3008b5 nodeName:}" failed. No retries permitted until 2026-09-08 16:30:50.139792429 +0000 UTC m=+34.302763855 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvcz8" (UniqueName: "kubernetes.io/projected/50d7b024-55d7-4077-8584-9657bf3008b5-kube-api-access-hvcz8") pod "network-check-target-mz7tj" (UID: "50d7b024-55d7-4077-8584-9657bf3008b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:30:34.340024 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:34.339943    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:34.340174 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:34.340079    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fm6rg" podUID="48a3c156-d793-4586-9ef9-c6770eb982d0"
Sep 08 16:30:35.340061 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:35.340024    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:35.340219 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:35.340071    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:35.340219 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:35.340159    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz7tj" podUID="50d7b024-55d7-4077-8584-9657bf3008b5"
Sep 08 16:30:35.340595 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:35.340297    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g67mm" podUID="0abcdd93-6ae1-4e51-a19b-911c06152822"
Sep 08 16:30:36.340603 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:36.340567    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:36.341040 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:36.340710    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fm6rg" podUID="48a3c156-d793-4586-9ef9-c6770eb982d0"
Sep 08 16:30:37.339742 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.339568    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:37.339874 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.339588    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:37.339874 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:37.339853    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g67mm" podUID="0abcdd93-6ae1-4e51-a19b-911c06152822"
Sep 08 16:30:37.340002 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:37.339908    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz7tj" podUID="50d7b024-55d7-4077-8584-9657bf3008b5"
Sep 08 16:30:37.440409 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.440374    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal" event={"ID":"555e95963eeb3a1ffd9bfc39675ae3f8","Type":"ContainerStarted","Data":"225ab7d5f49865023ac0f5a1928a435350dd1b1eb1b517349eb0b1e289afcc59"}
Sep 08 16:30:37.442035 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.442010    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rj64h" event={"ID":"ad2a7c50-5f92-449d-a650-ed26da4359b7","Type":"ContainerStarted","Data":"82354688e5810a01554bb1003623bf774e9436524cc92202fdcd2b1659ae7c6d"}
Sep 08 16:30:37.443572 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.443543    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dxt9v" event={"ID":"b925da05-aa52-48c7-8dc0-839d562e1d01","Type":"ContainerStarted","Data":"d60fe8847cc194ad42bf9d9e910636bee506aa58a1041b3bd46319a92635056c"}
Sep 08 16:30:37.445214 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.445193    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6" event={"ID":"3d1f5282-a130-4c07-86b0-c70f99879223","Type":"ContainerStarted","Data":"712ef26e03fed58c18f9c37cb3fb6de7c8e3169b3fe2baa71b06f200ac66b36b"}
Sep 08 16:30:37.448103 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.448085    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd" event={"ID":"41e9b515-3988-4240-a220-e3325ecfbcee","Type":"ContainerStarted","Data":"d7b866714dc07a3b7368b2fd26844e0320a701bd6ecb6b206faf01e40dec8d30"}
Sep 08 16:30:37.448202 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.448170    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd" event={"ID":"41e9b515-3988-4240-a220-e3325ecfbcee","Type":"ContainerStarted","Data":"9b0ebd77a509431aeed04331c526679ec9608acc2f4cb32dd52456a2f4e27bc2"}
Sep 08 16:30:37.448246 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.448235    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd" event={"ID":"41e9b515-3988-4240-a220-e3325ecfbcee","Type":"ContainerStarted","Data":"8ea405193d3b205b231821323cc05d2cdec4d37f138805008c589bec81317e6b"}
Sep 08 16:30:37.448293 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.448266    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd" event={"ID":"41e9b515-3988-4240-a220-e3325ecfbcee","Type":"ContainerStarted","Data":"8ae281b98aa79c4640501c2a6f709c9b33bc37cd053b008a5857cc3829a7aea3"}
Sep 08 16:30:37.448344 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.448303    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd" event={"ID":"41e9b515-3988-4240-a220-e3325ecfbcee","Type":"ContainerStarted","Data":"c562efd3e9cda30f212f4a9bcbeb732fbf7cce57efe0494104fcf0b15924d0d3"}
Sep 08 16:30:37.448344 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.448332    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd" event={"ID":"41e9b515-3988-4240-a220-e3325ecfbcee","Type":"ContainerStarted","Data":"a49841140dfa2652585a910a3cc312f29bfa88152752211ba1037e7dcd672ae5"}
Sep 08 16:30:37.449451 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.449431    2557 generic.go:356] "Generic (PLEG): container finished" podID="179f538010bbe319e093cd4c1c6661f9" containerID="ab9fd1f1f69ae3b24b86b41fd8de696365c9e4165db03bfe684dec99254cf025" exitCode=0
Sep 08 16:30:37.449532 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.449505    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" event={"ID":"179f538010bbe319e093cd4c1c6661f9","Type":"ContainerDied","Data":"ab9fd1f1f69ae3b24b86b41fd8de696365c9e4165db03bfe684dec99254cf025"}
Sep 08 16:30:37.450882 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.450859    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q8kt4" event={"ID":"ff68e8af-fa2a-4b97-a923-3db24f91d137","Type":"ContainerStarted","Data":"2b45ccd86a56097b442f7b3851bf3e13a49481f77885d920649fac443efab221"}
Sep 08 16:30:37.452476 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.452454    2557 generic.go:356] "Generic (PLEG): container finished" podID="10038ceb-d586-4fb2-99e2-dc326d9daa8c" containerID="8f5a2d8e9cb501195b147772089f2eba1c2bc3bd91b6baed056bbe159fe3f424" exitCode=0
Sep 08 16:30:37.452568 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.452521    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-62zlz" event={"ID":"10038ceb-d586-4fb2-99e2-dc326d9daa8c","Type":"ContainerDied","Data":"8f5a2d8e9cb501195b147772089f2eba1c2bc3bd91b6baed056bbe159fe3f424"}
Sep 08 16:30:37.454126 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.454103    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m4n88" event={"ID":"afe27658-50c9-47a8-884a-8ddc05f375ed","Type":"ContainerStarted","Data":"f1fc6c1c5a01331e6af9825bcf0845d933393f9e905cd0e98e9b1c3101bb741c"}
Sep 08 16:30:37.455689 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.455669    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pqhsk" event={"ID":"bd8c514b-7ca3-43e4-b87c-8a811c04efe8","Type":"ContainerStarted","Data":"3683e5edb54890914de54102416551a4dec00ae91aaa01de9165b77a5c32ce1b"}
Sep 08 16:30:37.456555 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.456520    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-51.ec2.internal" podStartSLOduration=20.45650458 podStartE2EDuration="20.45650458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:37.455913651 +0000 UTC m=+21.618885100" watchObservedRunningTime="2026-09-08 16:30:37.45650458 +0000 UTC m=+21.619476027"
Sep 08 16:30:37.515335 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.515287    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pqhsk" podStartSLOduration=7.245069171 podStartE2EDuration="21.515271788s" totalImagesPullingTime="14.270202617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:37.492382341 +0000 UTC m=+21.655353789" watchObservedRunningTime="2026-09-08 16:30:37.515271788 +0000 UTC m=+21.678243239"
Sep 08 16:30:37.531063 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.531021    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rj64h" podStartSLOduration=3.513922284 podStartE2EDuration="21.531006868s" totalImagesPullingTime="18.017084584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:37.530888267 +0000 UTC m=+21.693859714" watchObservedRunningTime="2026-09-08 16:30:37.531006868 +0000 UTC m=+21.693978315"
Sep 08 16:30:37.548763 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.548708    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-q8kt4" podStartSLOduration=3.324225566 podStartE2EDuration="21.548687727s" totalImagesPullingTime="18.224462161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:37.548229214 +0000 UTC m=+21.711200665" watchObservedRunningTime="2026-09-08 16:30:37.548687727 +0000 UTC m=+21.711659176"
Sep 08 16:30:37.563497 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.563437    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-m4n88" podStartSLOduration=7.437347274 podStartE2EDuration="21.563414773s" totalImagesPullingTime="14.126067499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:37.562964347 +0000 UTC m=+21.725935791" watchObservedRunningTime="2026-09-08 16:30:37.563414773 +0000 UTC m=+21.726386224"
Sep 08 16:30:37.583879 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:37.583535    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dxt9v" podStartSLOduration=3.53777193 podStartE2EDuration="21.583514394s" totalImagesPullingTime="18.045742464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:37.582609574 +0000 UTC m=+21.745581023" watchObservedRunningTime="2026-09-08 16:30:37.583514394 +0000 UTC m=+21.746485843"
Sep 08 16:30:38.340007 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:38.339721    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:38.340259 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:38.340060    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fm6rg" podUID="48a3c156-d793-4586-9ef9-c6770eb982d0"
Sep 08 16:30:38.459223 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:38.459189    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r4kbv" event={"ID":"d5c4bf16-72bf-41a9-bc8a-58cc50aa0cba","Type":"ContainerStarted","Data":"ff805d1c4e6155d769786814beff4d199bf3b4636709d9f783a1d39008a41219"}
Sep 08 16:30:38.461033 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:38.460993    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" event={"ID":"179f538010bbe319e093cd4c1c6661f9","Type":"ContainerStarted","Data":"6616cb11cf63c990896fe3d08855d2c20aa781a5a7370d8991fbc42d15d63565"}
Sep 08 16:30:38.475341 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:38.475287    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-r4kbv" podStartSLOduration=8.369078258 podStartE2EDuration="22.475267397s" totalImagesPullingTime="14.106189139s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:38.474567999 +0000 UTC m=+22.637539447" watchObservedRunningTime="2026-09-08 16:30:38.475267397 +0000 UTC m=+22.638238846"
Sep 08 16:30:38.489412 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:38.489369    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-51.ec2.internal" podStartSLOduration=21.489354355 podStartE2EDuration="21.489354355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:38.489292629 +0000 UTC m=+22.652264077" watchObservedRunningTime="2026-09-08 16:30:38.489354355 +0000 UTC m=+22.652325804"
Sep 08 16:30:38.757697 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:38.757662    2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 16:30:39.285062 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:39.284968    2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T16:30:38.757678846Z","UUID":"a78e47c0-995f-46cc-afd9-6acb671d3912","Handler":null,"Name":"","Endpoint":""}
Sep 08 16:30:39.286316 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:39.286298    2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 16:30:39.286431 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:39.286333    2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 16:30:39.340273 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:39.340234    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:39.340273 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:39.340271    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:39.340492 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:39.340362    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz7tj" podUID="50d7b024-55d7-4077-8584-9657bf3008b5"
Sep 08 16:30:39.340546 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:39.340489    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g67mm" podUID="0abcdd93-6ae1-4e51-a19b-911c06152822"
Sep 08 16:30:39.466313 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:39.466278    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6" event={"ID":"3d1f5282-a130-4c07-86b0-c70f99879223","Type":"ContainerStarted","Data":"82348b09f895523bac149e2f65b3cbc040828a800f26c633deabc6551a122851"}
Sep 08 16:30:39.472384 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:39.472346    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd" event={"ID":"41e9b515-3988-4240-a220-e3325ecfbcee","Type":"ContainerStarted","Data":"5403861400d8cae45fffbc9101905fe05bdd245fa122881e2c1ac817b6f51b10"}
Sep 08 16:30:40.339409 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:40.339325    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:40.339563 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:40.339474    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fm6rg" podUID="48a3c156-d793-4586-9ef9-c6770eb982d0"
Sep 08 16:30:40.344566 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:40.344508    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqhsk_bd8c514b-7ca3-43e4-b87c-8a811c04efe8/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:40.424770 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:40.424735    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pqhsk"
Sep 08 16:30:40.425497 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:40.425471    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pqhsk"
Sep 08 16:30:40.463204 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:40.463173    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqhsk_bd8c514b-7ca3-43e4-b87c-8a811c04efe8/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:40.476649 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:40.476621    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6" event={"ID":"3d1f5282-a130-4c07-86b0-c70f99879223","Type":"ContainerStarted","Data":"965a787e27bea7ee9ab6b7a895f56bb0ea41a97da9451b65132e98ab19661e8a"}
Sep 08 16:30:40.477074 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:40.476820    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pqhsk"
Sep 08 16:30:40.477266 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:40.477250    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pqhsk"
Sep 08 16:30:40.495877 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:40.495815    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrmb6" podStartSLOduration=3.688144087 podStartE2EDuration="24.495801204s" totalImagesPullingTime="20.807657117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:30:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:40.495531228 +0000 UTC m=+24.658502676" watchObservedRunningTime="2026-09-08 16:30:40.495801204 +0000 UTC m=+24.658772651"
Sep 08 16:30:40.567884 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:40.567856    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqhsk_bd8c514b-7ca3-43e4-b87c-8a811c04efe8/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:41.340322 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:41.340286    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:41.340322 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:41.340305    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:41.340549 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:41.340486    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g67mm" podUID="0abcdd93-6ae1-4e51-a19b-911c06152822"
Sep 08 16:30:41.340604 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:41.340547    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz7tj" podUID="50d7b024-55d7-4077-8584-9657bf3008b5"
Sep 08 16:30:41.779894 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:41.779857    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqhsk_bd8c514b-7ca3-43e4-b87c-8a811c04efe8/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:41.878123 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:41.878077    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqhsk_bd8c514b-7ca3-43e4-b87c-8a811c04efe8/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:42.339586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:42.339554    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:42.339753 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:42.339695    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fm6rg" podUID="48a3c156-d793-4586-9ef9-c6770eb982d0"
Sep 08 16:30:43.340136 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:43.340058    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:43.340528 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:43.340058    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:43.340528 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:43.340165    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz7tj" podUID="50d7b024-55d7-4077-8584-9657bf3008b5"
Sep 08 16:30:43.340528 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:43.340276    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g67mm" podUID="0abcdd93-6ae1-4e51-a19b-911c06152822"
Sep 08 16:30:43.489214 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:43.489183    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd" event={"ID":"41e9b515-3988-4240-a220-e3325ecfbcee","Type":"ContainerStarted","Data":"d2c7ebd767bfd303efbf7da09eb5902837239bc33ff210f8e93fd253829ca028"}
Sep 08 16:30:43.489794 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:43.489767    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:43.506226 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:43.506198    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:43.519529 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:43.519467    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd" podStartSLOduration=9.40794683 podStartE2EDuration="27.51945238s" totalImagesPullingTime="18.11150555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:43.518982512 +0000 UTC m=+27.681953960" watchObservedRunningTime="2026-09-08 16:30:43.51945238 +0000 UTC m=+27.682423827"
Sep 08 16:30:44.339722 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:44.339685    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:44.339960 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:44.339813    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fm6rg" podUID="48a3c156-d793-4586-9ef9-c6770eb982d0"
Sep 08 16:30:44.482906 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:44.482867    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqhsk_bd8c514b-7ca3-43e4-b87c-8a811c04efe8/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:44.496608 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:44.496571    2557 generic.go:356] "Generic (PLEG): container finished" podID="10038ceb-d586-4fb2-99e2-dc326d9daa8c" containerID="a1538baaaf668533f852fe6ffbf4448e93aca9767d558398bcb4ab7883000a69" exitCode=0
Sep 08 16:30:44.496763 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:44.496654    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-62zlz" event={"ID":"10038ceb-d586-4fb2-99e2-dc326d9daa8c","Type":"ContainerDied","Data":"a1538baaaf668533f852fe6ffbf4448e93aca9767d558398bcb4ab7883000a69"}
Sep 08 16:30:44.497176 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:44.497151    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:44.497281 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:44.497236    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:44.513698 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:44.513670    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:30:44.549893 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:44.549865    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqhsk_bd8c514b-7ca3-43e4-b87c-8a811c04efe8/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:45.015488 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:45.015405    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mz7tj"]
Sep 08 16:30:45.015638 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:45.015538    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:45.015638 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:45.015619    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz7tj" podUID="50d7b024-55d7-4077-8584-9657bf3008b5"
Sep 08 16:30:45.020962 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:45.020924    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g67mm"]
Sep 08 16:30:45.021104 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:45.021038    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:45.021156 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:45.021130    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g67mm" podUID="0abcdd93-6ae1-4e51-a19b-911c06152822"
Sep 08 16:30:45.021559 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:45.021534    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fm6rg"]
Sep 08 16:30:45.021665 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:45.021626    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:45.021775 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:45.021742    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fm6rg" podUID="48a3c156-d793-4586-9ef9-c6770eb982d0"
Sep 08 16:30:46.340792 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:46.340567    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:46.341323 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:46.340936    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fm6rg" podUID="48a3c156-d793-4586-9ef9-c6770eb982d0"
Sep 08 16:30:46.503401 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:46.503362    2557 generic.go:356] "Generic (PLEG): container finished" podID="10038ceb-d586-4fb2-99e2-dc326d9daa8c" containerID="c0c2df833e6dc086eae649ac58612a9196dbfab52c07ce3e55e69b020a143e63" exitCode=0
Sep 08 16:30:46.503582 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:46.503462    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-62zlz" event={"ID":"10038ceb-d586-4fb2-99e2-dc326d9daa8c","Type":"ContainerDied","Data":"c0c2df833e6dc086eae649ac58612a9196dbfab52c07ce3e55e69b020a143e63"}
Sep 08 16:30:47.339317 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:47.339284    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:47.339470 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:47.339284    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:47.339470 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:47.339398    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz7tj" podUID="50d7b024-55d7-4077-8584-9657bf3008b5"
Sep 08 16:30:47.339470 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:47.339454    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g67mm" podUID="0abcdd93-6ae1-4e51-a19b-911c06152822"
Sep 08 16:30:48.343267 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:48.343179    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:48.343688 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:48.343306    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fm6rg" podUID="48a3c156-d793-4586-9ef9-c6770eb982d0"
Sep 08 16:30:48.509972 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:48.509937    2557 generic.go:356] "Generic (PLEG): container finished" podID="10038ceb-d586-4fb2-99e2-dc326d9daa8c" containerID="8c94f52c28c17c5d747929f953cd4c29526a59b493314875e92cc3ecb5af9678" exitCode=0
Sep 08 16:30:48.510153 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:48.509980    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-62zlz" event={"ID":"10038ceb-d586-4fb2-99e2-dc326d9daa8c","Type":"ContainerDied","Data":"8c94f52c28c17c5d747929f953cd4c29526a59b493314875e92cc3ecb5af9678"}
Sep 08 16:30:49.339590 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.339559    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:49.339747 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.339559    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:49.339747 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:49.339669    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mz7tj" podUID="50d7b024-55d7-4077-8584-9657bf3008b5"
Sep 08 16:30:49.339747 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:49.339718    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g67mm" podUID="0abcdd93-6ae1-4e51-a19b-911c06152822"
Sep 08 16:30:49.637150 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.637118    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-51.ec2.internal" event="NodeReady"
Sep 08 16:30:49.637594 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.637276    2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 16:30:49.706367 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.706324    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kbwhw"]
Sep 08 16:30:49.724471 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.724432    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-shbt6"]
Sep 08 16:30:49.724648 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.724579    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kbwhw"
Sep 08 16:30:49.729956 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.728410    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 16:30:49.729956 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.729126    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7zsf6\""
Sep 08 16:30:49.729956 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.729287    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 16:30:49.742629 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.742604    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"]
Sep 08 16:30:49.742754 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.742721    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-shbt6"
Sep 08 16:30:49.745611 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.745552    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-sqr8x\""
Sep 08 16:30:49.745986 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.745959    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 16:30:49.746096 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.746001    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 16:30:49.746354 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.746334    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 16:30:49.760978 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.760947    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-shbt6"]
Sep 08 16:30:49.761102 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.760994    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kbwhw"]
Sep 08 16:30:49.761102 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.761022    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"]
Sep 08 16:30:49.761185 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.761112    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:49.767154 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.767125    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 08 16:30:49.767418 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.767398    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 08 16:30:49.767554 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.767535    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-x6c7g\""
Sep 08 16:30:49.767643 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.767628    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 08 16:30:49.772680 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.772655    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 08 16:30:49.850009 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.849912    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-tls\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:49.850009 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.849967    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsx24\" (UniqueName: \"kubernetes.io/projected/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-kube-api-access-wsx24\") pod \"dns-default-kbwhw\" (UID: \"195311dd-1be6-4f3d-ba2f-c7cdc374e97d\") " pod="openshift-dns/dns-default-kbwhw"
Sep 08 16:30:49.850009 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.849991    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00fce104-5845-4461-8d09-b70ebad8c5b2-image-registry-private-configuration\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:49.850345 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.850014    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00fce104-5845-4461-8d09-b70ebad8c5b2-ca-trust-extracted\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:49.850345 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.850150    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00fce104-5845-4461-8d09-b70ebad8c5b2-installation-pull-secrets\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:49.850345 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.850196    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-certificates\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:49.850345 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.850223    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ac1828f-7510-4cfe-ad8f-19cc7f57d3df-cert\") pod \"ingress-canary-shbt6\" (UID: \"3ac1828f-7510-4cfe-ad8f-19cc7f57d3df\") " pod="openshift-ingress-canary/ingress-canary-shbt6"
Sep 08 16:30:49.850345 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.850267    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-metrics-tls\") pod \"dns-default-kbwhw\" (UID: \"195311dd-1be6-4f3d-ba2f-c7cdc374e97d\") " pod="openshift-dns/dns-default-kbwhw"
Sep 08 16:30:49.850345 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.850308    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-tmp-dir\") pod \"dns-default-kbwhw\" (UID: \"195311dd-1be6-4f3d-ba2f-c7cdc374e97d\") " pod="openshift-dns/dns-default-kbwhw"
Sep 08 16:30:49.850345 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.850342    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-bound-sa-token\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:49.850619 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.850358    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw6sv\" (UniqueName: \"kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-kube-api-access-qw6sv\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:49.850619 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.850414    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-config-volume\") pod \"dns-default-kbwhw\" (UID: \"195311dd-1be6-4f3d-ba2f-c7cdc374e97d\") " pod="openshift-dns/dns-default-kbwhw"
Sep 08 16:30:49.850619 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.850441    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00fce104-5845-4461-8d09-b70ebad8c5b2-trusted-ca\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:49.850619 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.850462    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl2pr\" (UniqueName: \"kubernetes.io/projected/3ac1828f-7510-4cfe-ad8f-19cc7f57d3df-kube-api-access-fl2pr\") pod \"ingress-canary-shbt6\" (UID: \"3ac1828f-7510-4cfe-ad8f-19cc7f57d3df\") " pod="openshift-ingress-canary/ingress-canary-shbt6"
Sep 08 16:30:49.951219 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.951183    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48a3c156-d793-4586-9ef9-c6770eb982d0-metrics-certs\") pod \"network-metrics-daemon-fm6rg\" (UID: \"48a3c156-d793-4586-9ef9-c6770eb982d0\") " pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:49.951219 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.951222    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00fce104-5845-4461-8d09-b70ebad8c5b2-installation-pull-secrets\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:49.951462 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.951239    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-certificates\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:49.951462 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:49.951346    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:30:49.951462 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.951380    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ac1828f-7510-4cfe-ad8f-19cc7f57d3df-cert\") pod \"ingress-canary-shbt6\" (UID: \"3ac1828f-7510-4cfe-ad8f-19cc7f57d3df\") " pod="openshift-ingress-canary/ingress-canary-shbt6"
Sep 08 16:30:49.951462 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:49.951420    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48a3c156-d793-4586-9ef9-c6770eb982d0-metrics-certs podName:48a3c156-d793-4586-9ef9-c6770eb982d0 nodeName:}" failed. No retries permitted until 2026-09-08 16:31:21.951397753 +0000 UTC m=+66.114369196 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48a3c156-d793-4586-9ef9-c6770eb982d0-metrics-certs") pod "network-metrics-daemon-fm6rg" (UID: "48a3c156-d793-4586-9ef9-c6770eb982d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:30:49.951660 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.951461    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-metrics-tls\") pod \"dns-default-kbwhw\" (UID: \"195311dd-1be6-4f3d-ba2f-c7cdc374e97d\") " pod="openshift-dns/dns-default-kbwhw"
Sep 08 16:30:49.951660 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.951495    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-tmp-dir\") pod \"dns-default-kbwhw\" (UID: \"195311dd-1be6-4f3d-ba2f-c7cdc374e97d\") " pod="openshift-dns/dns-default-kbwhw"
Sep 08 16:30:49.951660 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.951536    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-bound-sa-token\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:49.951660 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:49.951555    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:30:49.951660 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:49.951626    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-metrics-tls podName:195311dd-1be6-4f3d-ba2f-c7cdc374e97d nodeName:}" failed. No retries permitted until 2026-09-08 16:30:50.451607643 +0000 UTC m=+34.614579068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-metrics-tls") pod "dns-default-kbwhw" (UID: "195311dd-1be6-4f3d-ba2f-c7cdc374e97d") : secret "dns-default-metrics-tls" not found
Sep 08 16:30:49.951660 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.951559    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw6sv\" (UniqueName: \"kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-kube-api-access-qw6sv\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:49.951945 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.951695    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-config-volume\") pod \"dns-default-kbwhw\" (UID: \"195311dd-1be6-4f3d-ba2f-c7cdc374e97d\") " pod="openshift-dns/dns-default-kbwhw"
Sep 08 16:30:49.951945 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.951742    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00fce104-5845-4461-8d09-b70ebad8c5b2-trusted-ca\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:49.951945 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.951828    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-certificates\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:49.951945 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.951856    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-tmp-dir\") pod \"dns-default-kbwhw\" (UID: \"195311dd-1be6-4f3d-ba2f-c7cdc374e97d\") " pod="openshift-dns/dns-default-kbwhw"
Sep 08 16:30:49.951945 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.951895    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl2pr\" (UniqueName: \"kubernetes.io/projected/3ac1828f-7510-4cfe-ad8f-19cc7f57d3df-kube-api-access-fl2pr\") pod \"ingress-canary-shbt6\" (UID: \"3ac1828f-7510-4cfe-ad8f-19cc7f57d3df\") " pod="openshift-ingress-canary/ingress-canary-shbt6"
Sep 08 16:30:49.951945 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.951921    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-tls\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:49.952200 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.951958    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsx24\" (UniqueName: \"kubernetes.io/projected/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-kube-api-access-wsx24\") pod \"dns-default-kbwhw\" (UID: \"195311dd-1be6-4f3d-ba2f-c7cdc374e97d\") " pod="openshift-dns/dns-default-kbwhw"
Sep 08 16:30:49.952200 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.951987    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00fce104-5845-4461-8d09-b70ebad8c5b2-image-registry-private-configuration\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:49.952200 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.952013    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00fce104-5845-4461-8d09-b70ebad8c5b2-ca-trust-extracted\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:49.952318 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:49.952209    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 16:30:49.952318 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:49.952228    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d54d7dd4b-ccnlr: secret "image-registry-tls" not found
Sep 08 16:30:49.952318 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.952240    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-config-volume\") pod \"dns-default-kbwhw\" (UID: \"195311dd-1be6-4f3d-ba2f-c7cdc374e97d\") " pod="openshift-dns/dns-default-kbwhw"
Sep 08 16:30:49.952318 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:49.952291    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-tls podName:00fce104-5845-4461-8d09-b70ebad8c5b2 nodeName:}" failed. No retries permitted until 2026-09-08 16:30:50.452276356 +0000 UTC m=+34.615247786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-tls") pod "image-registry-7d54d7dd4b-ccnlr" (UID: "00fce104-5845-4461-8d09-b70ebad8c5b2") : secret "image-registry-tls" not found
Sep 08 16:30:49.952318 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.952302    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00fce104-5845-4461-8d09-b70ebad8c5b2-ca-trust-extracted\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:49.952576 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.952557    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00fce104-5845-4461-8d09-b70ebad8c5b2-trusted-ca\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:49.956108 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.956067    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00fce104-5845-4461-8d09-b70ebad8c5b2-image-registry-private-configuration\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:49.956108 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.956077    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00fce104-5845-4461-8d09-b70ebad8c5b2-installation-pull-secrets\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:49.956293 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.956120    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ac1828f-7510-4cfe-ad8f-19cc7f57d3df-cert\") pod \"ingress-canary-shbt6\" (UID: \"3ac1828f-7510-4cfe-ad8f-19cc7f57d3df\") " pod="openshift-ingress-canary/ingress-canary-shbt6"
Sep 08 16:30:49.961164 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.960997    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl2pr\" (UniqueName: \"kubernetes.io/projected/3ac1828f-7510-4cfe-ad8f-19cc7f57d3df-kube-api-access-fl2pr\") pod \"ingress-canary-shbt6\" (UID: \"3ac1828f-7510-4cfe-ad8f-19cc7f57d3df\") " pod="openshift-ingress-canary/ingress-canary-shbt6"
Sep 08 16:30:49.961164 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.961150    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-bound-sa-token\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:49.961327 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.961203    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsx24\" (UniqueName: \"kubernetes.io/projected/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-kube-api-access-wsx24\") pod \"dns-default-kbwhw\" (UID: \"195311dd-1be6-4f3d-ba2f-c7cdc374e97d\") " pod="openshift-dns/dns-default-kbwhw"
Sep 08 16:30:49.961883 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:49.961828    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw6sv\" (UniqueName: \"kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-kube-api-access-qw6sv\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:50.052344 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:50.052308    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0abcdd93-6ae1-4e51-a19b-911c06152822-original-pull-secret\") pod \"global-pull-secret-syncer-g67mm\" (UID: \"0abcdd93-6ae1-4e51-a19b-911c06152822\") " pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:50.052496 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:50.052466    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:30:50.052553 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:50.052543    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0abcdd93-6ae1-4e51-a19b-911c06152822-original-pull-secret podName:0abcdd93-6ae1-4e51-a19b-911c06152822 nodeName:}" failed. No retries permitted until 2026-09-08 16:31:22.05252681 +0000 UTC m=+66.215498249 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0abcdd93-6ae1-4e51-a19b-911c06152822-original-pull-secret") pod "global-pull-secret-syncer-g67mm" (UID: "0abcdd93-6ae1-4e51-a19b-911c06152822") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:30:50.053236 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:50.053218    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-shbt6"
Sep 08 16:30:50.096303 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:50.096274    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqhsk_bd8c514b-7ca3-43e4-b87c-8a811c04efe8/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:50.153858 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:50.153734    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvcz8\" (UniqueName: \"kubernetes.io/projected/50d7b024-55d7-4077-8584-9657bf3008b5-kube-api-access-hvcz8\") pod \"network-check-target-mz7tj\" (UID: \"50d7b024-55d7-4077-8584-9657bf3008b5\") " pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:50.154475 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:50.154452    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:30:50.154475 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:50.154478    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:30:50.154639 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:50.154490    2557 projected.go:196] Error preparing data for projected volume kube-api-access-hvcz8 for pod openshift-network-diagnostics/network-check-target-mz7tj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:30:50.154639 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:50.154556    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50d7b024-55d7-4077-8584-9657bf3008b5-kube-api-access-hvcz8 podName:50d7b024-55d7-4077-8584-9657bf3008b5 nodeName:}" failed. No retries permitted until 2026-09-08 16:31:22.154537347 +0000 UTC m=+66.317508779 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hvcz8" (UniqueName: "kubernetes.io/projected/50d7b024-55d7-4077-8584-9657bf3008b5-kube-api-access-hvcz8") pod "network-check-target-mz7tj" (UID: "50d7b024-55d7-4077-8584-9657bf3008b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:30:50.343818 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:50.343790    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:30:50.346171 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:50.346147    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s6b8s\""
Sep 08 16:30:50.346292 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:50.346191    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 16:30:50.364927 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:50.364890    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqhsk_bd8c514b-7ca3-43e4-b87c-8a811c04efe8/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:50.378589 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:50.378561    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-shbt6"]
Sep 08 16:30:50.382184 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:50.382158    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac1828f_7510_4cfe_ad8f_19cc7f57d3df.slice/crio-7ff0267627ec563773f5af7965ab8d2826560a0c6458243e92dac3b16671d097 WatchSource:0}: Error finding container 7ff0267627ec563773f5af7965ab8d2826560a0c6458243e92dac3b16671d097: Status 404 returned error can't find the container with id 7ff0267627ec563773f5af7965ab8d2826560a0c6458243e92dac3b16671d097
Sep 08 16:30:50.456875 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:50.456816    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-tls\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:50.457063 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:50.456920    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-metrics-tls\") pod \"dns-default-kbwhw\" (UID: \"195311dd-1be6-4f3d-ba2f-c7cdc374e97d\") " pod="openshift-dns/dns-default-kbwhw"
Sep 08 16:30:50.457063 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:50.456963    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 16:30:50.457063 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:50.456985    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d54d7dd4b-ccnlr: secret "image-registry-tls" not found
Sep 08 16:30:50.457063 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:50.457023    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:30:50.457256 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:50.457069    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-tls podName:00fce104-5845-4461-8d09-b70ebad8c5b2 nodeName:}" failed. No retries permitted until 2026-09-08 16:30:51.457048274 +0000 UTC m=+35.620019720 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-tls") pod "image-registry-7d54d7dd4b-ccnlr" (UID: "00fce104-5845-4461-8d09-b70ebad8c5b2") : secret "image-registry-tls" not found
Sep 08 16:30:50.457256 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:50.457087    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-metrics-tls podName:195311dd-1be6-4f3d-ba2f-c7cdc374e97d nodeName:}" failed. No retries permitted until 2026-09-08 16:30:51.457078718 +0000 UTC m=+35.620050148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-metrics-tls") pod "dns-default-kbwhw" (UID: "195311dd-1be6-4f3d-ba2f-c7cdc374e97d") : secret "dns-default-metrics-tls" not found
Sep 08 16:30:50.515602 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:50.515563    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-shbt6" event={"ID":"3ac1828f-7510-4cfe-ad8f-19cc7f57d3df","Type":"ContainerStarted","Data":"7ff0267627ec563773f5af7965ab8d2826560a0c6458243e92dac3b16671d097"}
Sep 08 16:30:51.331503 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:51.331335    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqhsk_bd8c514b-7ca3-43e4-b87c-8a811c04efe8/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:51.339355 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:51.339325    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:30:51.339519 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:51.339503    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:30:51.342325 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:51.342306    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 16:30:51.342465 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:51.342306    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 16:30:51.343412 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:51.343049    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 16:30:51.343412 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:51.343267    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-86r89\""
Sep 08 16:30:51.414671 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:51.414634    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqhsk_bd8c514b-7ca3-43e4-b87c-8a811c04efe8/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:51.464628 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:51.464589    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-tls\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:51.464818 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:51.464664    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-metrics-tls\") pod \"dns-default-kbwhw\" (UID: \"195311dd-1be6-4f3d-ba2f-c7cdc374e97d\") " pod="openshift-dns/dns-default-kbwhw"
Sep 08 16:30:51.464818 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:51.464732    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 16:30:51.464818 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:51.464752    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d54d7dd4b-ccnlr: secret "image-registry-tls" not found
Sep 08 16:30:51.464818 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:51.464765    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:30:51.464818 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:51.464810    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-metrics-tls podName:195311dd-1be6-4f3d-ba2f-c7cdc374e97d nodeName:}" failed. No retries permitted until 2026-09-08 16:30:53.464793184 +0000 UTC m=+37.627764610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-metrics-tls") pod "dns-default-kbwhw" (UID: "195311dd-1be6-4f3d-ba2f-c7cdc374e97d") : secret "dns-default-metrics-tls" not found
Sep 08 16:30:51.465076 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:51.464855    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-tls podName:00fce104-5845-4461-8d09-b70ebad8c5b2 nodeName:}" failed. No retries permitted until 2026-09-08 16:30:53.464819731 +0000 UTC m=+37.627791159 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-tls") pod "image-registry-7d54d7dd4b-ccnlr" (UID: "00fce104-5845-4461-8d09-b70ebad8c5b2") : secret "image-registry-tls" not found
Sep 08 16:30:52.443608 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.443569    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqhsk_bd8c514b-7ca3-43e4-b87c-8a811c04efe8/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:52.444328 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.444302    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7qclj"]
Sep 08 16:30:52.454801 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.454777    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:30:52.457629 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.457575    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 16:30:52.457748 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.457677    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 16:30:52.458045 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.458027    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 16:30:52.458151 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.458050    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 16:30:52.459681 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.459662    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mpsgj\""
Sep 08 16:30:52.463898 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.463875    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7qclj"]
Sep 08 16:30:52.576234 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.576202    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/478b2574-9f73-4791-8411-90135387272a-crio-socket\") pod \"insights-runtime-extractor-7qclj\" (UID: \"478b2574-9f73-4791-8411-90135387272a\") " pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:30:52.576413 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.576327    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml4f6\" (UniqueName: \"kubernetes.io/projected/478b2574-9f73-4791-8411-90135387272a-kube-api-access-ml4f6\") pod \"insights-runtime-extractor-7qclj\" (UID: \"478b2574-9f73-4791-8411-90135387272a\") " pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:30:52.576413 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.576378    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/478b2574-9f73-4791-8411-90135387272a-data-volume\") pod \"insights-runtime-extractor-7qclj\" (UID: \"478b2574-9f73-4791-8411-90135387272a\") " pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:30:52.576502 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.576438    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/478b2574-9f73-4791-8411-90135387272a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qclj\" (UID: \"478b2574-9f73-4791-8411-90135387272a\") " pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:30:52.576502 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.576476    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/478b2574-9f73-4791-8411-90135387272a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qclj\" (UID: \"478b2574-9f73-4791-8411-90135387272a\") " pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:30:52.585208 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.585148    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqhsk_bd8c514b-7ca3-43e4-b87c-8a811c04efe8/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:52.676990 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.676953    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/478b2574-9f73-4791-8411-90135387272a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qclj\" (UID: \"478b2574-9f73-4791-8411-90135387272a\") " pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:30:52.677166 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.677034    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/478b2574-9f73-4791-8411-90135387272a-crio-socket\") pod \"insights-runtime-extractor-7qclj\" (UID: \"478b2574-9f73-4791-8411-90135387272a\") " pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:30:52.677166 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:52.677071    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 16:30:52.677166 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.677103    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml4f6\" (UniqueName: \"kubernetes.io/projected/478b2574-9f73-4791-8411-90135387272a-kube-api-access-ml4f6\") pod \"insights-runtime-extractor-7qclj\" (UID: \"478b2574-9f73-4791-8411-90135387272a\") " pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:30:52.677166 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:52.677158    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/478b2574-9f73-4791-8411-90135387272a-kube-rbac-proxy-cm podName:478b2574-9f73-4791-8411-90135387272a nodeName:}" failed. No retries permitted until 2026-09-08 16:30:53.17713872 +0000 UTC m=+37.340110165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/478b2574-9f73-4791-8411-90135387272a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7qclj" (UID: "478b2574-9f73-4791-8411-90135387272a") : configmap "kube-rbac-proxy" not found
Sep 08 16:30:52.677308 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.677178    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/478b2574-9f73-4791-8411-90135387272a-data-volume\") pod \"insights-runtime-extractor-7qclj\" (UID: \"478b2574-9f73-4791-8411-90135387272a\") " pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:30:52.677308 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.677218    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/478b2574-9f73-4791-8411-90135387272a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qclj\" (UID: \"478b2574-9f73-4791-8411-90135387272a\") " pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:30:52.677308 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:52.677300    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 16:30:52.677396 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:52.677335    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/478b2574-9f73-4791-8411-90135387272a-insights-runtime-extractor-tls podName:478b2574-9f73-4791-8411-90135387272a nodeName:}" failed. No retries permitted until 2026-09-08 16:30:53.177324112 +0000 UTC m=+37.340295538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/478b2574-9f73-4791-8411-90135387272a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7qclj" (UID: "478b2574-9f73-4791-8411-90135387272a") : secret "insights-runtime-extractor-tls" not found
Sep 08 16:30:52.677714 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.677692    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/478b2574-9f73-4791-8411-90135387272a-crio-socket\") pod \"insights-runtime-extractor-7qclj\" (UID: \"478b2574-9f73-4791-8411-90135387272a\") " pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:30:52.677750 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.677706    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/478b2574-9f73-4791-8411-90135387272a-data-volume\") pod \"insights-runtime-extractor-7qclj\" (UID: \"478b2574-9f73-4791-8411-90135387272a\") " pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:30:52.687384 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.687355    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml4f6\" (UniqueName: \"kubernetes.io/projected/478b2574-9f73-4791-8411-90135387272a-kube-api-access-ml4f6\") pod \"insights-runtime-extractor-7qclj\" (UID: \"478b2574-9f73-4791-8411-90135387272a\") " pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:30:52.708910 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.708796    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqhsk_bd8c514b-7ca3-43e4-b87c-8a811c04efe8/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:52.809399 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.809359    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqhsk_bd8c514b-7ca3-43e4-b87c-8a811c04efe8/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:52.938713 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.938676    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqhsk_bd8c514b-7ca3-43e4-b87c-8a811c04efe8/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:52.968344 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.968273    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-55s6v"]
Sep 08 16:30:52.979828 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.979804    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-55s6v"
Sep 08 16:30:52.986882 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.986618    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 08 16:30:52.986882 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.986668    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8jwgs\""
Sep 08 16:30:52.987047 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.987035    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 08 16:30:52.987237 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:52.987219    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-55s6v"]
Sep 08 16:30:53.043709 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:53.043672    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqhsk_bd8c514b-7ca3-43e4-b87c-8a811c04efe8/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:53.081161 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:53.081117    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbb52\" (UniqueName: \"kubernetes.io/projected/0fb47bbb-ca12-44e0-b61e-76a3196c05e6-kube-api-access-dbb52\") pod \"migrator-fdbb8fdff-55s6v\" (UID: \"0fb47bbb-ca12-44e0-b61e-76a3196c05e6\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-55s6v"
Sep 08 16:30:53.145574 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:53.145530    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqhsk_bd8c514b-7ca3-43e4-b87c-8a811c04efe8/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:53.182098 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:53.182060    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/478b2574-9f73-4791-8411-90135387272a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qclj\" (UID: \"478b2574-9f73-4791-8411-90135387272a\") " pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:30:53.182279 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:53.182109    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/478b2574-9f73-4791-8411-90135387272a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qclj\" (UID: \"478b2574-9f73-4791-8411-90135387272a\") " pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:30:53.182279 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:53.182170    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbb52\" (UniqueName: \"kubernetes.io/projected/0fb47bbb-ca12-44e0-b61e-76a3196c05e6-kube-api-access-dbb52\") pod \"migrator-fdbb8fdff-55s6v\" (UID: \"0fb47bbb-ca12-44e0-b61e-76a3196c05e6\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-55s6v"
Sep 08 16:30:53.182279 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:53.182231    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 16:30:53.182440 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:53.182276    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 16:30:53.182440 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:53.182307    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/478b2574-9f73-4791-8411-90135387272a-insights-runtime-extractor-tls podName:478b2574-9f73-4791-8411-90135387272a nodeName:}" failed. No retries permitted until 2026-09-08 16:30:54.182285419 +0000 UTC m=+38.345256849 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/478b2574-9f73-4791-8411-90135387272a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7qclj" (UID: "478b2574-9f73-4791-8411-90135387272a") : secret "insights-runtime-extractor-tls" not found
Sep 08 16:30:53.182440 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:53.182349    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/478b2574-9f73-4791-8411-90135387272a-kube-rbac-proxy-cm podName:478b2574-9f73-4791-8411-90135387272a nodeName:}" failed. No retries permitted until 2026-09-08 16:30:54.182330711 +0000 UTC m=+38.345302150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/478b2574-9f73-4791-8411-90135387272a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7qclj" (UID: "478b2574-9f73-4791-8411-90135387272a") : configmap "kube-rbac-proxy" not found
Sep 08 16:30:53.191494 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:53.191466    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbb52\" (UniqueName: \"kubernetes.io/projected/0fb47bbb-ca12-44e0-b61e-76a3196c05e6-kube-api-access-dbb52\") pod \"migrator-fdbb8fdff-55s6v\" (UID: \"0fb47bbb-ca12-44e0-b61e-76a3196c05e6\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-55s6v"
Sep 08 16:30:53.288935 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:53.288861    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-55s6v"
Sep 08 16:30:53.434704 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:53.434667    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqhsk_bd8c514b-7ca3-43e4-b87c-8a811c04efe8/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:53.484537 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:53.484506    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-tls\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:53.485107 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:53.484562    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-metrics-tls\") pod \"dns-default-kbwhw\" (UID: \"195311dd-1be6-4f3d-ba2f-c7cdc374e97d\") " pod="openshift-dns/dns-default-kbwhw"
Sep 08 16:30:53.485107 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:53.484673    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:30:53.485107 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:53.484681    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 16:30:53.485107 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:53.484705    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d54d7dd4b-ccnlr: secret "image-registry-tls" not found
Sep 08 16:30:53.485107 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:53.484755    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-metrics-tls podName:195311dd-1be6-4f3d-ba2f-c7cdc374e97d nodeName:}" failed. No retries permitted until 2026-09-08 16:30:57.484737505 +0000 UTC m=+41.647708932 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-metrics-tls") pod "dns-default-kbwhw" (UID: "195311dd-1be6-4f3d-ba2f-c7cdc374e97d") : secret "dns-default-metrics-tls" not found
Sep 08 16:30:53.485107 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:53.484773    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-tls podName:00fce104-5845-4461-8d09-b70ebad8c5b2 nodeName:}" failed. No retries permitted until 2026-09-08 16:30:57.484766651 +0000 UTC m=+41.647738076 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-tls") pod "image-registry-7d54d7dd4b-ccnlr" (UID: "00fce104-5845-4461-8d09-b70ebad8c5b2") : secret "image-registry-tls" not found
Sep 08 16:30:53.546403 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:53.546302    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pqhsk_bd8c514b-7ca3-43e4-b87c-8a811c04efe8/konnectivity-agent/0.log" limit=1024
Sep 08 16:30:53.994139 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:53.994112    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rj64h_ad2a7c50-5f92-449d-a650-ed26da4359b7/dns-node-resolver/0.log"
Sep 08 16:30:54.192088 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:54.192051    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/478b2574-9f73-4791-8411-90135387272a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qclj\" (UID: \"478b2574-9f73-4791-8411-90135387272a\") " pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:30:54.192088 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:54.192102    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/478b2574-9f73-4791-8411-90135387272a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qclj\" (UID: \"478b2574-9f73-4791-8411-90135387272a\") " pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:30:54.192299 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:54.192175    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 16:30:54.192299 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:54.192214    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 16:30:54.192299 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:54.192241    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/478b2574-9f73-4791-8411-90135387272a-kube-rbac-proxy-cm podName:478b2574-9f73-4791-8411-90135387272a nodeName:}" failed. No retries permitted until 2026-09-08 16:30:56.192221886 +0000 UTC m=+40.355193313 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/478b2574-9f73-4791-8411-90135387272a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7qclj" (UID: "478b2574-9f73-4791-8411-90135387272a") : configmap "kube-rbac-proxy" not found
Sep 08 16:30:54.192299 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:54.192267    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/478b2574-9f73-4791-8411-90135387272a-insights-runtime-extractor-tls podName:478b2574-9f73-4791-8411-90135387272a nodeName:}" failed. No retries permitted until 2026-09-08 16:30:56.192253048 +0000 UTC m=+40.355224473 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/478b2574-9f73-4791-8411-90135387272a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7qclj" (UID: "478b2574-9f73-4791-8411-90135387272a") : secret "insights-runtime-extractor-tls" not found
Sep 08 16:30:55.000337 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.000311    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m4n88_afe27658-50c9-47a8-884a-8ddc05f375ed/node-ca/0.log"
Sep 08 16:30:55.031328 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.031292    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-f58nb"]
Sep 08 16:30:55.054549 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.054478    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-f58nb"]
Sep 08 16:30:55.054684 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.054604    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-f58nb"
Sep 08 16:30:55.059970 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.059946    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 08 16:30:55.060899 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.060882    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 08 16:30:55.061154 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.061034    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 08 16:30:55.061154 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.061116    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 08 16:30:55.061327 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.061290    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-r6wls\""
Sep 08 16:30:55.066801 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.066779    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-55s6v"]
Sep 08 16:30:55.119707 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:55.119672    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb47bbb_ca12_44e0_b61e_76a3196c05e6.slice/crio-11b075c442c6b9e646b9994a19302b143ab12f761a9bae006082aa33916dd775 WatchSource:0}: Error finding container 11b075c442c6b9e646b9994a19302b143ab12f761a9bae006082aa33916dd775: Status 404 returned error can't find the container with id 11b075c442c6b9e646b9994a19302b143ab12f761a9bae006082aa33916dd775
Sep 08 16:30:55.199325 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.199284    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e6f06762-06c9-40f3-a348-0cf733bb80c8-signing-key\") pod \"service-ca-75d99c484-f58nb\" (UID: \"e6f06762-06c9-40f3-a348-0cf733bb80c8\") " pod="openshift-service-ca/service-ca-75d99c484-f58nb"
Sep 08 16:30:55.199485 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.199340    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2slk\" (UniqueName: \"kubernetes.io/projected/e6f06762-06c9-40f3-a348-0cf733bb80c8-kube-api-access-b2slk\") pod \"service-ca-75d99c484-f58nb\" (UID: \"e6f06762-06c9-40f3-a348-0cf733bb80c8\") " pod="openshift-service-ca/service-ca-75d99c484-f58nb"
Sep 08 16:30:55.199485 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.199379    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e6f06762-06c9-40f3-a348-0cf733bb80c8-signing-cabundle\") pod \"service-ca-75d99c484-f58nb\" (UID: \"e6f06762-06c9-40f3-a348-0cf733bb80c8\") " pod="openshift-service-ca/service-ca-75d99c484-f58nb"
Sep 08 16:30:55.300128 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.300086    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e6f06762-06c9-40f3-a348-0cf733bb80c8-signing-key\") pod \"service-ca-75d99c484-f58nb\" (UID: \"e6f06762-06c9-40f3-a348-0cf733bb80c8\") " pod="openshift-service-ca/service-ca-75d99c484-f58nb"
Sep 08 16:30:55.300311 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.300153    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2slk\" (UniqueName: \"kubernetes.io/projected/e6f06762-06c9-40f3-a348-0cf733bb80c8-kube-api-access-b2slk\") pod \"service-ca-75d99c484-f58nb\" (UID: \"e6f06762-06c9-40f3-a348-0cf733bb80c8\") " pod="openshift-service-ca/service-ca-75d99c484-f58nb"
Sep 08 16:30:55.300311 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.300204    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e6f06762-06c9-40f3-a348-0cf733bb80c8-signing-cabundle\") pod \"service-ca-75d99c484-f58nb\" (UID: \"e6f06762-06c9-40f3-a348-0cf733bb80c8\") " pod="openshift-service-ca/service-ca-75d99c484-f58nb"
Sep 08 16:30:55.300824 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.300800    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e6f06762-06c9-40f3-a348-0cf733bb80c8-signing-cabundle\") pod \"service-ca-75d99c484-f58nb\" (UID: \"e6f06762-06c9-40f3-a348-0cf733bb80c8\") " pod="openshift-service-ca/service-ca-75d99c484-f58nb"
Sep 08 16:30:55.302604 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.302586    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e6f06762-06c9-40f3-a348-0cf733bb80c8-signing-key\") pod \"service-ca-75d99c484-f58nb\" (UID: \"e6f06762-06c9-40f3-a348-0cf733bb80c8\") " pod="openshift-service-ca/service-ca-75d99c484-f58nb"
Sep 08 16:30:55.309653 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.309610    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2slk\" (UniqueName: \"kubernetes.io/projected/e6f06762-06c9-40f3-a348-0cf733bb80c8-kube-api-access-b2slk\") pod \"service-ca-75d99c484-f58nb\" (UID: \"e6f06762-06c9-40f3-a348-0cf733bb80c8\") " pod="openshift-service-ca/service-ca-75d99c484-f58nb"
Sep 08 16:30:55.365724 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.365695    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-f58nb"
Sep 08 16:30:55.528874 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.528818    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-55s6v" event={"ID":"0fb47bbb-ca12-44e0-b61e-76a3196c05e6","Type":"ContainerStarted","Data":"11b075c442c6b9e646b9994a19302b143ab12f761a9bae006082aa33916dd775"}
Sep 08 16:30:55.530340 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.530317    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-shbt6" event={"ID":"3ac1828f-7510-4cfe-ad8f-19cc7f57d3df","Type":"ContainerStarted","Data":"d5e47389c13423d926239a7cdf7edaf3c1b54d4eb9ccf1df7bc19d12b00b597d"}
Sep 08 16:30:55.533232 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.533197    2557 generic.go:356] "Generic (PLEG): container finished" podID="10038ceb-d586-4fb2-99e2-dc326d9daa8c" containerID="c272fea92c03e224b28bac6cbc1bac1d6d6b9cffa766862ea22a6564bb123aeb" exitCode=0
Sep 08 16:30:55.533334 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.533237    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-62zlz" event={"ID":"10038ceb-d586-4fb2-99e2-dc326d9daa8c","Type":"ContainerDied","Data":"c272fea92c03e224b28bac6cbc1bac1d6d6b9cffa766862ea22a6564bb123aeb"}
Sep 08 16:30:55.551378 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.551331    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-shbt6" podStartSLOduration=2.139616691 podStartE2EDuration="6.551315749s" totalImagesPullingTime="4.411699058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:55.550651303 +0000 UTC m=+39.713622750" watchObservedRunningTime="2026-09-08 16:30:55.551315749 +0000 UTC m=+39.714287196"
Sep 08 16:30:55.659210 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.658981    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-f58nb"]
Sep 08 16:30:55.661710 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:30:55.661680    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6f06762_06c9_40f3_a348_0cf733bb80c8.slice/crio-71c3bd37bd91cf7856bae3d5465bf69df6fb699765f385e0d8bded72f5ca8bcc WatchSource:0}: Error finding container 71c3bd37bd91cf7856bae3d5465bf69df6fb699765f385e0d8bded72f5ca8bcc: Status 404 returned error can't find the container with id 71c3bd37bd91cf7856bae3d5465bf69df6fb699765f385e0d8bded72f5ca8bcc
Sep 08 16:30:55.799543 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:55.799506    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-shbt6_3ac1828f-7510-4cfe-ad8f-19cc7f57d3df/serve-healthcheck-canary/0.log"
Sep 08 16:30:56.208677 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:56.208633    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/478b2574-9f73-4791-8411-90135387272a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qclj\" (UID: \"478b2574-9f73-4791-8411-90135387272a\") " pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:30:56.209171 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:56.208682    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/478b2574-9f73-4791-8411-90135387272a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qclj\" (UID: \"478b2574-9f73-4791-8411-90135387272a\") " pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:30:56.209171 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:56.208818    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 16:30:56.209171 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:56.208920    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/478b2574-9f73-4791-8411-90135387272a-insights-runtime-extractor-tls podName:478b2574-9f73-4791-8411-90135387272a nodeName:}" failed. No retries permitted until 2026-09-08 16:31:00.208896269 +0000 UTC m=+44.371867698 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/478b2574-9f73-4791-8411-90135387272a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7qclj" (UID: "478b2574-9f73-4791-8411-90135387272a") : secret "insights-runtime-extractor-tls" not found
Sep 08 16:30:56.209351 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:56.209329    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/478b2574-9f73-4791-8411-90135387272a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qclj\" (UID: \"478b2574-9f73-4791-8411-90135387272a\") " pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:30:56.537087 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:56.537006    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-f58nb" event={"ID":"e6f06762-06c9-40f3-a348-0cf733bb80c8","Type":"ContainerStarted","Data":"71c3bd37bd91cf7856bae3d5465bf69df6fb699765f385e0d8bded72f5ca8bcc"}
Sep 08 16:30:56.539969 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:56.539941    2557 generic.go:356] "Generic (PLEG): container finished" podID="10038ceb-d586-4fb2-99e2-dc326d9daa8c" containerID="d1468cbf7e029d2efd42231ce1d4aa9b5591e579f076a853a7e3ffe4d56438d4" exitCode=0
Sep 08 16:30:56.540091 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:56.540025    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-62zlz" event={"ID":"10038ceb-d586-4fb2-99e2-dc326d9daa8c","Type":"ContainerDied","Data":"d1468cbf7e029d2efd42231ce1d4aa9b5591e579f076a853a7e3ffe4d56438d4"}
Sep 08 16:30:57.520395 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:57.520355    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-metrics-tls\") pod \"dns-default-kbwhw\" (UID: \"195311dd-1be6-4f3d-ba2f-c7cdc374e97d\") " pod="openshift-dns/dns-default-kbwhw"
Sep 08 16:30:57.520828 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:57.520444    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-tls\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:30:57.520828 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:57.520524    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:30:57.520828 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:57.520556    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 16:30:57.520828 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:57.520567    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d54d7dd4b-ccnlr: secret "image-registry-tls" not found
Sep 08 16:30:57.520828 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:57.520604    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-metrics-tls podName:195311dd-1be6-4f3d-ba2f-c7cdc374e97d nodeName:}" failed. No retries permitted until 2026-09-08 16:31:05.520582201 +0000 UTC m=+49.683553629 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-metrics-tls") pod "dns-default-kbwhw" (UID: "195311dd-1be6-4f3d-ba2f-c7cdc374e97d") : secret "dns-default-metrics-tls" not found
Sep 08 16:30:57.520828 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:30:57.520624    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-tls podName:00fce104-5845-4461-8d09-b70ebad8c5b2 nodeName:}" failed. No retries permitted until 2026-09-08 16:31:05.520615373 +0000 UTC m=+49.683586806 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-tls") pod "image-registry-7d54d7dd4b-ccnlr" (UID: "00fce104-5845-4461-8d09-b70ebad8c5b2") : secret "image-registry-tls" not found
Sep 08 16:30:57.546744 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:57.546701    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-62zlz" event={"ID":"10038ceb-d586-4fb2-99e2-dc326d9daa8c","Type":"ContainerStarted","Data":"462694005b78f431b95eb0900afb2382459b121777dee731d0a5dea057af5c50"}
Sep 08 16:30:57.548853 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:57.548805    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-55s6v" event={"ID":"0fb47bbb-ca12-44e0-b61e-76a3196c05e6","Type":"ContainerStarted","Data":"6bd33850be4988777f4e0549b9fa399a339ad9ec81f83e0c1a8e6507b9429249"}
Sep 08 16:30:57.548977 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:57.548876    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-55s6v" event={"ID":"0fb47bbb-ca12-44e0-b61e-76a3196c05e6","Type":"ContainerStarted","Data":"b1710ec286d18b8c9f68fac9e0ba5839e756ed4d7fb8f1d2d9d4d71b8c6b3547"}
Sep 08 16:30:57.576327 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:57.576282    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-62zlz" podStartSLOduration=8.279039746 podStartE2EDuration="41.576262245s" totalImagesPullingTime="33.297222499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:30:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:57.574518918 +0000 UTC m=+41.737490389" watchObservedRunningTime="2026-09-08 16:30:57.576262245 +0000 UTC m=+41.739233694"
Sep 08 16:30:57.590025 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:57.589978    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-55s6v" podStartSLOduration=3.747730355 podStartE2EDuration="5.589934588s" totalImagesPullingTime="1.842204233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:57.589799003 +0000 UTC m=+41.752770451" watchObservedRunningTime="2026-09-08 16:30:57.589934588 +0000 UTC m=+41.752906038"
Sep 08 16:30:59.555467 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:59.555376    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-f58nb" event={"ID":"e6f06762-06c9-40f3-a348-0cf733bb80c8","Type":"ContainerStarted","Data":"fa44f302f037eb40b512486640ea0fa7f0a5c17e2f18175a3fdd4f3197b1629a"}
Sep 08 16:30:59.578633 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:30:59.578578    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-f58nb" podStartSLOduration=1.014701652 podStartE2EDuration="4.578557325s" totalImagesPullingTime="3.563855673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:59.577051802 +0000 UTC m=+43.740023262" watchObservedRunningTime="2026-09-08 16:30:59.578557325 +0000 UTC m=+43.741528774"
Sep 08 16:31:00.240329 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:00.240285    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/478b2574-9f73-4791-8411-90135387272a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qclj\" (UID: \"478b2574-9f73-4791-8411-90135387272a\") " pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:31:00.240559 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:31:00.240508    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 16:31:00.240673 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:31:00.240660    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/478b2574-9f73-4791-8411-90135387272a-insights-runtime-extractor-tls podName:478b2574-9f73-4791-8411-90135387272a nodeName:}" failed. No retries permitted until 2026-09-08 16:31:08.240639919 +0000 UTC m=+52.403611360 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/478b2574-9f73-4791-8411-90135387272a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7qclj" (UID: "478b2574-9f73-4791-8411-90135387272a") : secret "insights-runtime-extractor-tls" not found
Sep 08 16:31:05.585644 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:05.585600    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-tls\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:31:05.586139 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:05.585673    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-metrics-tls\") pod \"dns-default-kbwhw\" (UID: \"195311dd-1be6-4f3d-ba2f-c7cdc374e97d\") " pod="openshift-dns/dns-default-kbwhw"
Sep 08 16:31:05.588100 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:05.588076    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/195311dd-1be6-4f3d-ba2f-c7cdc374e97d-metrics-tls\") pod \"dns-default-kbwhw\" (UID: \"195311dd-1be6-4f3d-ba2f-c7cdc374e97d\") " pod="openshift-dns/dns-default-kbwhw"
Sep 08 16:31:05.588195 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:05.588125    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-tls\") pod \"image-registry-7d54d7dd4b-ccnlr\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") " pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:31:05.637328 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:05.637290    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kbwhw"
Sep 08 16:31:05.672530 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:05.672489    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:31:05.907856 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:05.907743    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kbwhw"]
Sep 08 16:31:05.912231 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:31:05.912202    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195311dd_1be6_4f3d_ba2f_c7cdc374e97d.slice/crio-d729a9dcc90784bcab1cb954ab016bddc5dec231d9eb38f821a599c955488b20 WatchSource:0}: Error finding container d729a9dcc90784bcab1cb954ab016bddc5dec231d9eb38f821a599c955488b20: Status 404 returned error can't find the container with id d729a9dcc90784bcab1cb954ab016bddc5dec231d9eb38f821a599c955488b20
Sep 08 16:31:05.951244 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:05.951213    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"]
Sep 08 16:31:05.954095 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:31:05.954066    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00fce104_5845_4461_8d09_b70ebad8c5b2.slice/crio-fcaf3b8d14ea8b2b9a6f3f4370a26d0c86fd26d86e11e2529ce5f9ba0b9b5d4e WatchSource:0}: Error finding container fcaf3b8d14ea8b2b9a6f3f4370a26d0c86fd26d86e11e2529ce5f9ba0b9b5d4e: Status 404 returned error can't find the container with id fcaf3b8d14ea8b2b9a6f3f4370a26d0c86fd26d86e11e2529ce5f9ba0b9b5d4e
Sep 08 16:31:06.573720 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:06.573684    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr" event={"ID":"00fce104-5845-4461-8d09-b70ebad8c5b2","Type":"ContainerStarted","Data":"941a761d29cb62e83833a9554a041be57ed7a7b5968beaf211b6c81e03d62d8e"}
Sep 08 16:31:06.573921 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:06.573732    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr" event={"ID":"00fce104-5845-4461-8d09-b70ebad8c5b2","Type":"ContainerStarted","Data":"fcaf3b8d14ea8b2b9a6f3f4370a26d0c86fd26d86e11e2529ce5f9ba0b9b5d4e"}
Sep 08 16:31:06.573921 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:06.573793    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:31:06.574882 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:06.574861    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kbwhw" event={"ID":"195311dd-1be6-4f3d-ba2f-c7cdc374e97d","Type":"ContainerStarted","Data":"d729a9dcc90784bcab1cb954ab016bddc5dec231d9eb38f821a599c955488b20"}
Sep 08 16:31:06.597272 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:06.597212    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr" podStartSLOduration=37.59719794 podStartE2EDuration="37.59719794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:31:06.595794358 +0000 UTC m=+50.758765808" watchObservedRunningTime="2026-09-08 16:31:06.59719794 +0000 UTC m=+50.760169388"
Sep 08 16:31:08.308165 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:08.308127    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/478b2574-9f73-4791-8411-90135387272a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qclj\" (UID: \"478b2574-9f73-4791-8411-90135387272a\") " pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:31:08.310930 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:08.310898    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/478b2574-9f73-4791-8411-90135387272a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qclj\" (UID: \"478b2574-9f73-4791-8411-90135387272a\") " pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:31:08.366071 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:08.366036    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7qclj"
Sep 08 16:31:08.733516 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:08.733484    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7qclj"]
Sep 08 16:31:08.737949 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:31:08.737924    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod478b2574_9f73_4791_8411_90135387272a.slice/crio-08e0244be5c111a6d6243430d1522b0971a09ec9e1d024f5d38c15ad12cf30da WatchSource:0}: Error finding container 08e0244be5c111a6d6243430d1522b0971a09ec9e1d024f5d38c15ad12cf30da: Status 404 returned error can't find the container with id 08e0244be5c111a6d6243430d1522b0971a09ec9e1d024f5d38c15ad12cf30da
Sep 08 16:31:09.586531 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:09.586499    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qclj" event={"ID":"478b2574-9f73-4791-8411-90135387272a","Type":"ContainerStarted","Data":"1481a8dab13c53afb8dc01164d2b3f6166c8faf778accf1b4917e5e50ab9d0bd"}
Sep 08 16:31:09.586901 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:09.586603    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qclj" event={"ID":"478b2574-9f73-4791-8411-90135387272a","Type":"ContainerStarted","Data":"2f13498e150e282d55b85888914f0f7c427371ced2a1082328d495f4c17c0a66"}
Sep 08 16:31:09.586901 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:09.586630    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qclj" event={"ID":"478b2574-9f73-4791-8411-90135387272a","Type":"ContainerStarted","Data":"08e0244be5c111a6d6243430d1522b0971a09ec9e1d024f5d38c15ad12cf30da"}
Sep 08 16:31:09.588678 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:09.588656    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kbwhw" event={"ID":"195311dd-1be6-4f3d-ba2f-c7cdc374e97d","Type":"ContainerStarted","Data":"7d888e48d32956d1f2691da79f1e54d2f793c514e1cc43f3049dc921c5fe0bae"}
Sep 08 16:31:09.588779 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:09.588697    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kbwhw" event={"ID":"195311dd-1be6-4f3d-ba2f-c7cdc374e97d","Type":"ContainerStarted","Data":"cf6a5cedc84e02c03c730be8324e6bae03a79174bbce29cd82f5e30f06fe285a"}
Sep 08 16:31:09.588820 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:09.588793    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kbwhw"
Sep 08 16:31:09.610181 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:09.610129    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kbwhw" podStartSLOduration=18.017191483 podStartE2EDuration="20.610114583s" totalImagesPullingTime="2.5929231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:31:09.609798509 +0000 UTC m=+53.772769968" watchObservedRunningTime="2026-09-08 16:31:09.610114583 +0000 UTC m=+53.773086078"
Sep 08 16:31:11.597631 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:11.597593    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qclj" event={"ID":"478b2574-9f73-4791-8411-90135387272a","Type":"ContainerStarted","Data":"c120810e391f9652f6218794e1d84f4bc8ebc8bf78aa219cf774b08e6d1dc145"}
Sep 08 16:31:11.621120 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:11.621071    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7qclj" podStartSLOduration=17.410903059 podStartE2EDuration="19.621054829s" totalImagesPullingTime="2.21015177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:30:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:31:11.619735247 +0000 UTC m=+55.782706697" watchObservedRunningTime="2026-09-08 16:31:11.621054829 +0000 UTC m=+55.784026277"
Sep 08 16:31:14.521933 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.521903    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69bbd779d5-f5dw8"]
Sep 08 16:31:14.524089 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.524055    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69bbd779d5-f5dw8"
Sep 08 16:31:14.524856 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.524823    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dc49f58-rdttc"]
Sep 08 16:31:14.526388 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.526372    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dc49f58-rdttc"
Sep 08 16:31:14.528217 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.528200    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 08 16:31:14.529196 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.529177    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 08 16:31:14.529306 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.529191    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 08 16:31:14.529658 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.529639    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 08 16:31:14.530468 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.530449    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m"]
Sep 08 16:31:14.531996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.531976    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 08 16:31:14.532875 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.532855    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m"
Sep 08 16:31:14.533038 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.533024    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-tbpnn\""
Sep 08 16:31:14.535521 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.535499    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 08 16:31:14.535616 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.535539    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 08 16:31:14.535616 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.535599    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 08 16:31:14.535824 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.535810    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 08 16:31:14.540685 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.540666    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69bbd779d5-f5dw8"]
Sep 08 16:31:14.541442 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.541424    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dc49f58-rdttc"]
Sep 08 16:31:14.553172 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.553151    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m"]
Sep 08 16:31:14.593100 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.593070    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"]
Sep 08 16:31:14.653382 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.653350    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e21df0b-aa24-4bc5-80c5-a471fe2b3754-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-869dc49f58-rdttc\" (UID: \"7e21df0b-aa24-4bc5-80c5-a471fe2b3754\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dc49f58-rdttc"
Sep 08 16:31:14.653382 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.653387    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp8fm\" (UniqueName: \"kubernetes.io/projected/7e21df0b-aa24-4bc5-80c5-a471fe2b3754-kube-api-access-sp8fm\") pod \"managed-serviceaccount-addon-agent-869dc49f58-rdttc\" (UID: \"7e21df0b-aa24-4bc5-80c5-a471fe2b3754\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dc49f58-rdttc"
Sep 08 16:31:14.653578 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.653413    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs2hx\" (UniqueName: \"kubernetes.io/projected/08a4f313-500f-449b-be68-d67cee3d7f91-kube-api-access-rs2hx\") pod \"klusterlet-addon-workmgr-69bbd779d5-f5dw8\" (UID: \"08a4f313-500f-449b-be68-d67cee3d7f91\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69bbd779d5-f5dw8"
Sep 08 16:31:14.653578 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.653490    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a6fb1d78-c691-4814-8bfb-2f910a646899-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85f76f5566-l4s6m\" (UID: \"a6fb1d78-c691-4814-8bfb-2f910a646899\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m"
Sep 08 16:31:14.653578 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.653528    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a6fb1d78-c691-4814-8bfb-2f910a646899-hub\") pod \"cluster-proxy-proxy-agent-85f76f5566-l4s6m\" (UID: \"a6fb1d78-c691-4814-8bfb-2f910a646899\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m"
Sep 08 16:31:14.653578 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.653572    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/08a4f313-500f-449b-be68-d67cee3d7f91-klusterlet-config\") pod \"klusterlet-addon-workmgr-69bbd779d5-f5dw8\" (UID: \"08a4f313-500f-449b-be68-d67cee3d7f91\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69bbd779d5-f5dw8"
Sep 08 16:31:14.653703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.653642    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6fb1d78-c691-4814-8bfb-2f910a646899-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85f76f5566-l4s6m\" (UID: \"a6fb1d78-c691-4814-8bfb-2f910a646899\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m"
Sep 08 16:31:14.653703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.653660    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn4hx\" (UniqueName: \"kubernetes.io/projected/a6fb1d78-c691-4814-8bfb-2f910a646899-kube-api-access-nn4hx\") pod \"cluster-proxy-proxy-agent-85f76f5566-l4s6m\" (UID: \"a6fb1d78-c691-4814-8bfb-2f910a646899\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m"
Sep 08 16:31:14.653703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.653682    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a6fb1d78-c691-4814-8bfb-2f910a646899-ca\") pod \"cluster-proxy-proxy-agent-85f76f5566-l4s6m\" (UID: \"a6fb1d78-c691-4814-8bfb-2f910a646899\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m"
Sep 08 16:31:14.653703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.653697    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08a4f313-500f-449b-be68-d67cee3d7f91-tmp\") pod \"klusterlet-addon-workmgr-69bbd779d5-f5dw8\" (UID: \"08a4f313-500f-449b-be68-d67cee3d7f91\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69bbd779d5-f5dw8"
Sep 08 16:31:14.653818 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.653736    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a6fb1d78-c691-4814-8bfb-2f910a646899-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85f76f5566-l4s6m\" (UID: \"a6fb1d78-c691-4814-8bfb-2f910a646899\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m"
Sep 08 16:31:14.658978 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.658953    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7967f76576-9zbkf"]
Sep 08 16:31:14.660914 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.660894    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.677934 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.677904    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7967f76576-9zbkf"]
Sep 08 16:31:14.754400 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.754354    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkzsq\" (UniqueName: \"kubernetes.io/projected/d92d5e13-8f2c-4850-a003-248b524e9b6b-kube-api-access-wkzsq\") pod \"image-registry-7967f76576-9zbkf\" (UID: \"d92d5e13-8f2c-4850-a003-248b524e9b6b\") " pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.754561 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.754419    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/08a4f313-500f-449b-be68-d67cee3d7f91-klusterlet-config\") pod \"klusterlet-addon-workmgr-69bbd779d5-f5dw8\" (UID: \"08a4f313-500f-449b-be68-d67cee3d7f91\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69bbd779d5-f5dw8"
Sep 08 16:31:14.754561 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.754466    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6fb1d78-c691-4814-8bfb-2f910a646899-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85f76f5566-l4s6m\" (UID: \"a6fb1d78-c691-4814-8bfb-2f910a646899\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m"
Sep 08 16:31:14.754561 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.754497    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn4hx\" (UniqueName: \"kubernetes.io/projected/a6fb1d78-c691-4814-8bfb-2f910a646899-kube-api-access-nn4hx\") pod \"cluster-proxy-proxy-agent-85f76f5566-l4s6m\" (UID: \"a6fb1d78-c691-4814-8bfb-2f910a646899\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m"
Sep 08 16:31:14.754561 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.754534    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a6fb1d78-c691-4814-8bfb-2f910a646899-ca\") pod \"cluster-proxy-proxy-agent-85f76f5566-l4s6m\" (UID: \"a6fb1d78-c691-4814-8bfb-2f910a646899\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m"
Sep 08 16:31:14.754732 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.754562    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d92d5e13-8f2c-4850-a003-248b524e9b6b-ca-trust-extracted\") pod \"image-registry-7967f76576-9zbkf\" (UID: \"d92d5e13-8f2c-4850-a003-248b524e9b6b\") " pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.754732 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.754587    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08a4f313-500f-449b-be68-d67cee3d7f91-tmp\") pod \"klusterlet-addon-workmgr-69bbd779d5-f5dw8\" (UID: \"08a4f313-500f-449b-be68-d67cee3d7f91\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69bbd779d5-f5dw8"
Sep 08 16:31:14.754732 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.754683    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d92d5e13-8f2c-4850-a003-248b524e9b6b-bound-sa-token\") pod \"image-registry-7967f76576-9zbkf\" (UID: \"d92d5e13-8f2c-4850-a003-248b524e9b6b\") " pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.754889 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.754740    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a6fb1d78-c691-4814-8bfb-2f910a646899-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85f76f5566-l4s6m\" (UID: \"a6fb1d78-c691-4814-8bfb-2f910a646899\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m"
Sep 08 16:31:14.754889 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.754783    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d92d5e13-8f2c-4850-a003-248b524e9b6b-installation-pull-secrets\") pod \"image-registry-7967f76576-9zbkf\" (UID: \"d92d5e13-8f2c-4850-a003-248b524e9b6b\") " pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.754889 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.754811    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e21df0b-aa24-4bc5-80c5-a471fe2b3754-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-869dc49f58-rdttc\" (UID: \"7e21df0b-aa24-4bc5-80c5-a471fe2b3754\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dc49f58-rdttc"
Sep 08 16:31:14.755054 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.754943    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp8fm\" (UniqueName: \"kubernetes.io/projected/7e21df0b-aa24-4bc5-80c5-a471fe2b3754-kube-api-access-sp8fm\") pod \"managed-serviceaccount-addon-agent-869dc49f58-rdttc\" (UID: \"7e21df0b-aa24-4bc5-80c5-a471fe2b3754\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dc49f58-rdttc"
Sep 08 16:31:14.755054 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.754967    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08a4f313-500f-449b-be68-d67cee3d7f91-tmp\") pod \"klusterlet-addon-workmgr-69bbd779d5-f5dw8\" (UID: \"08a4f313-500f-449b-be68-d67cee3d7f91\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69bbd779d5-f5dw8"
Sep 08 16:31:14.755054 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.754984    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs2hx\" (UniqueName: \"kubernetes.io/projected/08a4f313-500f-449b-be68-d67cee3d7f91-kube-api-access-rs2hx\") pod \"klusterlet-addon-workmgr-69bbd779d5-f5dw8\" (UID: \"08a4f313-500f-449b-be68-d67cee3d7f91\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69bbd779d5-f5dw8"
Sep 08 16:31:14.755203 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.755053    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d92d5e13-8f2c-4850-a003-248b524e9b6b-registry-tls\") pod \"image-registry-7967f76576-9zbkf\" (UID: \"d92d5e13-8f2c-4850-a003-248b524e9b6b\") " pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.755203 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.755104    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a6fb1d78-c691-4814-8bfb-2f910a646899-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85f76f5566-l4s6m\" (UID: \"a6fb1d78-c691-4814-8bfb-2f910a646899\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m"
Sep 08 16:31:14.755203 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.755146    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d92d5e13-8f2c-4850-a003-248b524e9b6b-registry-certificates\") pod \"image-registry-7967f76576-9zbkf\" (UID: \"d92d5e13-8f2c-4850-a003-248b524e9b6b\") " pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.755568 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.755362    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a6fb1d78-c691-4814-8bfb-2f910a646899-hub\") pod \"cluster-proxy-proxy-agent-85f76f5566-l4s6m\" (UID: \"a6fb1d78-c691-4814-8bfb-2f910a646899\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m"
Sep 08 16:31:14.755568 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.755411    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d92d5e13-8f2c-4850-a003-248b524e9b6b-trusted-ca\") pod \"image-registry-7967f76576-9zbkf\" (UID: \"d92d5e13-8f2c-4850-a003-248b524e9b6b\") " pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.755568 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.755443    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d92d5e13-8f2c-4850-a003-248b524e9b6b-image-registry-private-configuration\") pod \"image-registry-7967f76576-9zbkf\" (UID: \"d92d5e13-8f2c-4850-a003-248b524e9b6b\") " pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.756590 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.756316    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6fb1d78-c691-4814-8bfb-2f910a646899-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85f76f5566-l4s6m\" (UID: \"a6fb1d78-c691-4814-8bfb-2f910a646899\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m"
Sep 08 16:31:14.757529 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.757497    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/08a4f313-500f-449b-be68-d67cee3d7f91-klusterlet-config\") pod \"klusterlet-addon-workmgr-69bbd779d5-f5dw8\" (UID: \"08a4f313-500f-449b-be68-d67cee3d7f91\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69bbd779d5-f5dw8"
Sep 08 16:31:14.757718 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.757683    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a6fb1d78-c691-4814-8bfb-2f910a646899-ca\") pod \"cluster-proxy-proxy-agent-85f76f5566-l4s6m\" (UID: \"a6fb1d78-c691-4814-8bfb-2f910a646899\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m"
Sep 08 16:31:14.757941 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.757911    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a6fb1d78-c691-4814-8bfb-2f910a646899-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85f76f5566-l4s6m\" (UID: \"a6fb1d78-c691-4814-8bfb-2f910a646899\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m"
Sep 08 16:31:14.759273 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.759248    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e21df0b-aa24-4bc5-80c5-a471fe2b3754-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-869dc49f58-rdttc\" (UID: \"7e21df0b-aa24-4bc5-80c5-a471fe2b3754\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dc49f58-rdttc"
Sep 08 16:31:14.760041 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.759773    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a6fb1d78-c691-4814-8bfb-2f910a646899-hub\") pod \"cluster-proxy-proxy-agent-85f76f5566-l4s6m\" (UID: \"a6fb1d78-c691-4814-8bfb-2f910a646899\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m"
Sep 08 16:31:14.763637 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.763618    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a6fb1d78-c691-4814-8bfb-2f910a646899-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85f76f5566-l4s6m\" (UID: \"a6fb1d78-c691-4814-8bfb-2f910a646899\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m"
Sep 08 16:31:14.776989 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.776926    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs2hx\" (UniqueName: \"kubernetes.io/projected/08a4f313-500f-449b-be68-d67cee3d7f91-kube-api-access-rs2hx\") pod \"klusterlet-addon-workmgr-69bbd779d5-f5dw8\" (UID: \"08a4f313-500f-449b-be68-d67cee3d7f91\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69bbd779d5-f5dw8"
Sep 08 16:31:14.776989 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.776931    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn4hx\" (UniqueName: \"kubernetes.io/projected/a6fb1d78-c691-4814-8bfb-2f910a646899-kube-api-access-nn4hx\") pod \"cluster-proxy-proxy-agent-85f76f5566-l4s6m\" (UID: \"a6fb1d78-c691-4814-8bfb-2f910a646899\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m"
Sep 08 16:31:14.776989 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.776978    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp8fm\" (UniqueName: \"kubernetes.io/projected/7e21df0b-aa24-4bc5-80c5-a471fe2b3754-kube-api-access-sp8fm\") pod \"managed-serviceaccount-addon-agent-869dc49f58-rdttc\" (UID: \"7e21df0b-aa24-4bc5-80c5-a471fe2b3754\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dc49f58-rdttc"
Sep 08 16:31:14.834574 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.834536    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69bbd779d5-f5dw8"
Sep 08 16:31:14.854048 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.854026    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dc49f58-rdttc"
Sep 08 16:31:14.857373 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.857346    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d92d5e13-8f2c-4850-a003-248b524e9b6b-installation-pull-secrets\") pod \"image-registry-7967f76576-9zbkf\" (UID: \"d92d5e13-8f2c-4850-a003-248b524e9b6b\") " pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.857479 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.857397    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d92d5e13-8f2c-4850-a003-248b524e9b6b-registry-tls\") pod \"image-registry-7967f76576-9zbkf\" (UID: \"d92d5e13-8f2c-4850-a003-248b524e9b6b\") " pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.857479 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.857441    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d92d5e13-8f2c-4850-a003-248b524e9b6b-registry-certificates\") pod \"image-registry-7967f76576-9zbkf\" (UID: \"d92d5e13-8f2c-4850-a003-248b524e9b6b\") " pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.857479 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.857462    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d92d5e13-8f2c-4850-a003-248b524e9b6b-trusted-ca\") pod \"image-registry-7967f76576-9zbkf\" (UID: \"d92d5e13-8f2c-4850-a003-248b524e9b6b\") " pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.857659 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.857484    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d92d5e13-8f2c-4850-a003-248b524e9b6b-image-registry-private-configuration\") pod \"image-registry-7967f76576-9zbkf\" (UID: \"d92d5e13-8f2c-4850-a003-248b524e9b6b\") " pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.857659 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.857520    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkzsq\" (UniqueName: \"kubernetes.io/projected/d92d5e13-8f2c-4850-a003-248b524e9b6b-kube-api-access-wkzsq\") pod \"image-registry-7967f76576-9zbkf\" (UID: \"d92d5e13-8f2c-4850-a003-248b524e9b6b\") " pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.857659 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.857584    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d92d5e13-8f2c-4850-a003-248b524e9b6b-ca-trust-extracted\") pod \"image-registry-7967f76576-9zbkf\" (UID: \"d92d5e13-8f2c-4850-a003-248b524e9b6b\") " pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.857659 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.857615    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d92d5e13-8f2c-4850-a003-248b524e9b6b-bound-sa-token\") pod \"image-registry-7967f76576-9zbkf\" (UID: \"d92d5e13-8f2c-4850-a003-248b524e9b6b\") " pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.858489 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.858466    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m"
Sep 08 16:31:14.858666 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.858596    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d92d5e13-8f2c-4850-a003-248b524e9b6b-registry-certificates\") pod \"image-registry-7967f76576-9zbkf\" (UID: \"d92d5e13-8f2c-4850-a003-248b524e9b6b\") " pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.858779 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.858696    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d92d5e13-8f2c-4850-a003-248b524e9b6b-ca-trust-extracted\") pod \"image-registry-7967f76576-9zbkf\" (UID: \"d92d5e13-8f2c-4850-a003-248b524e9b6b\") " pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.859712 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.859685    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d92d5e13-8f2c-4850-a003-248b524e9b6b-trusted-ca\") pod \"image-registry-7967f76576-9zbkf\" (UID: \"d92d5e13-8f2c-4850-a003-248b524e9b6b\") " pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.860125 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.860104    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d92d5e13-8f2c-4850-a003-248b524e9b6b-image-registry-private-configuration\") pod \"image-registry-7967f76576-9zbkf\" (UID: \"d92d5e13-8f2c-4850-a003-248b524e9b6b\") " pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.860444 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.860424    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d92d5e13-8f2c-4850-a003-248b524e9b6b-installation-pull-secrets\") pod \"image-registry-7967f76576-9zbkf\" (UID: \"d92d5e13-8f2c-4850-a003-248b524e9b6b\") " pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.860918 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.860897    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d92d5e13-8f2c-4850-a003-248b524e9b6b-registry-tls\") pod \"image-registry-7967f76576-9zbkf\" (UID: \"d92d5e13-8f2c-4850-a003-248b524e9b6b\") " pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.870724 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.870701    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d92d5e13-8f2c-4850-a003-248b524e9b6b-bound-sa-token\") pod \"image-registry-7967f76576-9zbkf\" (UID: \"d92d5e13-8f2c-4850-a003-248b524e9b6b\") " pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.870796 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.870745    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkzsq\" (UniqueName: \"kubernetes.io/projected/d92d5e13-8f2c-4850-a003-248b524e9b6b-kube-api-access-wkzsq\") pod \"image-registry-7967f76576-9zbkf\" (UID: \"d92d5e13-8f2c-4850-a003-248b524e9b6b\") " pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:14.979154 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:14.979121    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:15.113948 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:15.113914    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69bbd779d5-f5dw8"]
Sep 08 16:31:15.117532 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:31:15.117506    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a4f313_500f_449b_be68_d67cee3d7f91.slice/crio-de3216b3c88ed80ea887f6ba209c8af1d5f85e63da1a56a10e173eae6441fb6c WatchSource:0}: Error finding container de3216b3c88ed80ea887f6ba209c8af1d5f85e63da1a56a10e173eae6441fb6c: Status 404 returned error can't find the container with id de3216b3c88ed80ea887f6ba209c8af1d5f85e63da1a56a10e173eae6441fb6c
Sep 08 16:31:15.147100 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:15.147068    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dc49f58-rdttc"]
Sep 08 16:31:15.150078 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:31:15.150047    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e21df0b_aa24_4bc5_80c5_a471fe2b3754.slice/crio-89f1dcf1fb3d2883370c9b087530ea32f6460b3f80fd278fb0eb4cf51c0135cf WatchSource:0}: Error finding container 89f1dcf1fb3d2883370c9b087530ea32f6460b3f80fd278fb0eb4cf51c0135cf: Status 404 returned error can't find the container with id 89f1dcf1fb3d2883370c9b087530ea32f6460b3f80fd278fb0eb4cf51c0135cf
Sep 08 16:31:15.152181 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:15.152104    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m"]
Sep 08 16:31:15.155326 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:31:15.155306    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6fb1d78_c691_4814_8bfb_2f910a646899.slice/crio-76f4e875410fdf7f61311077405dd751da1eeb73a9a2b6e2af742a8d40027fcf WatchSource:0}: Error finding container 76f4e875410fdf7f61311077405dd751da1eeb73a9a2b6e2af742a8d40027fcf: Status 404 returned error can't find the container with id 76f4e875410fdf7f61311077405dd751da1eeb73a9a2b6e2af742a8d40027fcf
Sep 08 16:31:15.258402 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:15.258364    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7967f76576-9zbkf"]
Sep 08 16:31:15.261091 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:31:15.261065    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd92d5e13_8f2c_4850_a003_248b524e9b6b.slice/crio-37e9f8d5e069bbed18c14e1a2f7d861ac3aa3c45cc08639b2b6621ad36169ade WatchSource:0}: Error finding container 37e9f8d5e069bbed18c14e1a2f7d861ac3aa3c45cc08639b2b6621ad36169ade: Status 404 returned error can't find the container with id 37e9f8d5e069bbed18c14e1a2f7d861ac3aa3c45cc08639b2b6621ad36169ade
Sep 08 16:31:15.612447 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:15.612407    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7967f76576-9zbkf" event={"ID":"d92d5e13-8f2c-4850-a003-248b524e9b6b","Type":"ContainerStarted","Data":"fc15a310bbb8ab5b294051558cfece44aeb46d5d2299d748af4907895886e73d"}
Sep 08 16:31:15.612899 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:15.612476    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7967f76576-9zbkf" event={"ID":"d92d5e13-8f2c-4850-a003-248b524e9b6b","Type":"ContainerStarted","Data":"37e9f8d5e069bbed18c14e1a2f7d861ac3aa3c45cc08639b2b6621ad36169ade"}
Sep 08 16:31:15.612899 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:15.612504    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:15.613637 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:15.613609    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m" event={"ID":"a6fb1d78-c691-4814-8bfb-2f910a646899","Type":"ContainerStarted","Data":"76f4e875410fdf7f61311077405dd751da1eeb73a9a2b6e2af742a8d40027fcf"}
Sep 08 16:31:15.614661 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:15.614636    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69bbd779d5-f5dw8" event={"ID":"08a4f313-500f-449b-be68-d67cee3d7f91","Type":"ContainerStarted","Data":"de3216b3c88ed80ea887f6ba209c8af1d5f85e63da1a56a10e173eae6441fb6c"}
Sep 08 16:31:15.615710 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:15.615683    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dc49f58-rdttc" event={"ID":"7e21df0b-aa24-4bc5-80c5-a471fe2b3754","Type":"ContainerStarted","Data":"89f1dcf1fb3d2883370c9b087530ea32f6460b3f80fd278fb0eb4cf51c0135cf"}
Sep 08 16:31:15.636396 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:15.636350    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7967f76576-9zbkf" podStartSLOduration=1.636334133 podStartE2EDuration="1.636334133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:31:15.634918539 +0000 UTC m=+59.797889998" watchObservedRunningTime="2026-09-08 16:31:15.636334133 +0000 UTC m=+59.799305582"
Sep 08 16:31:16.524884 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:16.524847    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mfqgd"
Sep 08 16:31:19.596002 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:19.595958    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kbwhw"
Sep 08 16:31:22.017551 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.017521    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48a3c156-d793-4586-9ef9-c6770eb982d0-metrics-certs\") pod \"network-metrics-daemon-fm6rg\" (UID: \"48a3c156-d793-4586-9ef9-c6770eb982d0\") " pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:31:22.020030 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.020007    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 16:31:22.030125 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.030099    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48a3c156-d793-4586-9ef9-c6770eb982d0-metrics-certs\") pod \"network-metrics-daemon-fm6rg\" (UID: \"48a3c156-d793-4586-9ef9-c6770eb982d0\") " pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:31:22.118783 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.118745    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0abcdd93-6ae1-4e51-a19b-911c06152822-original-pull-secret\") pod \"global-pull-secret-syncer-g67mm\" (UID: \"0abcdd93-6ae1-4e51-a19b-911c06152822\") " pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:31:22.121734 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.121712    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 16:31:22.132196 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.132176    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0abcdd93-6ae1-4e51-a19b-911c06152822-original-pull-secret\") pod \"global-pull-secret-syncer-g67mm\" (UID: \"0abcdd93-6ae1-4e51-a19b-911c06152822\") " pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:31:22.158157 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.158127    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s6b8s\""
Sep 08 16:31:22.166352 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.166318    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fm6rg"
Sep 08 16:31:22.219258 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.219223    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvcz8\" (UniqueName: \"kubernetes.io/projected/50d7b024-55d7-4077-8584-9657bf3008b5-kube-api-access-hvcz8\") pod \"network-check-target-mz7tj\" (UID: \"50d7b024-55d7-4077-8584-9657bf3008b5\") " pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:31:22.221893 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.221871    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 16:31:22.233445 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.233416    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 16:31:22.243519 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.243459    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvcz8\" (UniqueName: \"kubernetes.io/projected/50d7b024-55d7-4077-8584-9657bf3008b5-kube-api-access-hvcz8\") pod \"network-check-target-mz7tj\" (UID: \"50d7b024-55d7-4077-8584-9657bf3008b5\") " pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:31:22.254433 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.254405    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-86r89\""
Sep 08 16:31:22.259495 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.259471    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g67mm"
Sep 08 16:31:22.262777 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.262756    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:31:22.438046 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.438016    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fm6rg"]
Sep 08 16:31:22.441203 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:31:22.441175    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a3c156_d793_4586_9ef9_c6770eb982d0.slice/crio-3bd0f90783572eddc7d0ec56b385217893c43974ddabfc30967faeab4bb38e95 WatchSource:0}: Error finding container 3bd0f90783572eddc7d0ec56b385217893c43974ddabfc30967faeab4bb38e95: Status 404 returned error can't find the container with id 3bd0f90783572eddc7d0ec56b385217893c43974ddabfc30967faeab4bb38e95
Sep 08 16:31:22.534883 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.534861    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mz7tj"]
Sep 08 16:31:22.537099 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.537072    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g67mm"]
Sep 08 16:31:22.537482 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:31:22.537378    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d7b024_55d7_4077_8584_9657bf3008b5.slice/crio-7d59d43efc8f47d74464492b1aadea2b44c089572a65384d0d18ac57cc8d4ea1 WatchSource:0}: Error finding container 7d59d43efc8f47d74464492b1aadea2b44c089572a65384d0d18ac57cc8d4ea1: Status 404 returned error can't find the container with id 7d59d43efc8f47d74464492b1aadea2b44c089572a65384d0d18ac57cc8d4ea1
Sep 08 16:31:22.539780 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:31:22.539762    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0abcdd93_6ae1_4e51_a19b_911c06152822.slice/crio-eee5a3ac7dc575748d38410e4a5ecbaf2665b00baca2b8edc7ab0d18881a7d91 WatchSource:0}: Error finding container eee5a3ac7dc575748d38410e4a5ecbaf2665b00baca2b8edc7ab0d18881a7d91: Status 404 returned error can't find the container with id eee5a3ac7dc575748d38410e4a5ecbaf2665b00baca2b8edc7ab0d18881a7d91
Sep 08 16:31:22.649287 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.649247    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69bbd779d5-f5dw8" event={"ID":"08a4f313-500f-449b-be68-d67cee3d7f91","Type":"ContainerStarted","Data":"87069c90e84de45ffe8857626cfa7315ed3e939dade26792e9758e2289444c1a"}
Sep 08 16:31:22.649486 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.649438    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69bbd779d5-f5dw8"
Sep 08 16:31:22.650611 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.650575    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g67mm" event={"ID":"0abcdd93-6ae1-4e51-a19b-911c06152822","Type":"ContainerStarted","Data":"eee5a3ac7dc575748d38410e4a5ecbaf2665b00baca2b8edc7ab0d18881a7d91"}
Sep 08 16:31:22.651306 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.651285    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69bbd779d5-f5dw8"
Sep 08 16:31:22.651991 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.651971    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fm6rg" event={"ID":"48a3c156-d793-4586-9ef9-c6770eb982d0","Type":"ContainerStarted","Data":"3bd0f90783572eddc7d0ec56b385217893c43974ddabfc30967faeab4bb38e95"}
Sep 08 16:31:22.653297 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.653275    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dc49f58-rdttc" event={"ID":"7e21df0b-aa24-4bc5-80c5-a471fe2b3754","Type":"ContainerStarted","Data":"49bbaf6c7f4a7723fb6506a0bdb2e3badeb315c985711f221c300425d7c1238a"}
Sep 08 16:31:22.654546 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.654526    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mz7tj" event={"ID":"50d7b024-55d7-4077-8584-9657bf3008b5","Type":"ContainerStarted","Data":"7d59d43efc8f47d74464492b1aadea2b44c089572a65384d0d18ac57cc8d4ea1"}
Sep 08 16:31:22.656571 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.656553    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m" event={"ID":"a6fb1d78-c691-4814-8bfb-2f910a646899","Type":"ContainerStarted","Data":"68738613a23b9a7e2d11bd0177811da860ab5684e5cc52e5d320e2a7ffd722c4"}
Sep 08 16:31:22.656656 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.656580    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m" event={"ID":"a6fb1d78-c691-4814-8bfb-2f910a646899","Type":"ContainerStarted","Data":"e06af6a97b26e1c76d0374dc2cae332c5aa8630f79f571880f3df0a9c9d8e68b"}
Sep 08 16:31:22.656656 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.656592    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m" event={"ID":"a6fb1d78-c691-4814-8bfb-2f910a646899","Type":"ContainerStarted","Data":"99558fdd95d969457ac66708d6051e339e3389b6ae295009c645f9f31dc1de5d"}
Sep 08 16:31:22.672626 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.672586    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69bbd779d5-f5dw8" podStartSLOduration=2.006458994 podStartE2EDuration="8.672571986s" totalImagesPullingTime="6.666112992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:31:22.671821916 +0000 UTC m=+66.834793364" watchObservedRunningTime="2026-09-08 16:31:22.672571986 +0000 UTC m=+66.835543435"
Sep 08 16:31:22.714346 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.714293    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dc49f58-rdttc" podStartSLOduration=2.10395263 podStartE2EDuration="8.714275799s" totalImagesPullingTime="6.610323169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:31:22.691497134 +0000 UTC m=+66.854468584" watchObservedRunningTime="2026-09-08 16:31:22.714275799 +0000 UTC m=+66.877247246"
Sep 08 16:31:22.735055 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:22.735003    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85f76f5566-l4s6m" podStartSLOduration=2.109079842 podStartE2EDuration="8.734986526s" totalImagesPullingTime="6.625906684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:31:22.733581208 +0000 UTC m=+66.896552657" watchObservedRunningTime="2026-09-08 16:31:22.734986526 +0000 UTC m=+66.897957975"
Sep 08 16:31:23.664247 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:23.664207    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fm6rg" event={"ID":"48a3c156-d793-4586-9ef9-c6770eb982d0","Type":"ContainerStarted","Data":"36fbc9a42b2b4b0dc0524bd8033f332208da37382f695ce7188d0edb61239da7"}
Sep 08 16:31:24.600257 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:24.600213    2557 patch_prober.go:28] interesting pod/image-registry-7d54d7dd4b-ccnlr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 08 16:31:24.600421 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:24.600290    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr" podUID="00fce104-5845-4461-8d09-b70ebad8c5b2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 16:31:24.670178 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:24.670134    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fm6rg" event={"ID":"48a3c156-d793-4586-9ef9-c6770eb982d0","Type":"ContainerStarted","Data":"25e103da60ab2c5afda0b8114ed2f386dbe872b618e5d93465e1c7c25940399b"}
Sep 08 16:31:24.686623 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:24.686565    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fm6rg" podStartSLOduration=67.599914852 podStartE2EDuration="1m8.686549335s" totalImagesPullingTime="1.086634483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:31:24.686091827 +0000 UTC m=+68.849063275" watchObservedRunningTime="2026-09-08 16:31:24.686549335 +0000 UTC m=+68.849520783"
Sep 08 16:31:27.947797 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:27.947754    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2wnvq"]
Sep 08 16:31:27.973057 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:27.973027    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:27.977544 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:27.977355    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 16:31:27.978757 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:27.978367    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 16:31:27.978757 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:27.978431    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 16:31:27.978757 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:27.978636    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 16:31:27.978757 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:27.978654    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lhmx8\""
Sep 08 16:31:27.979054 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:27.978892    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 16:31:27.979106 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:27.979084    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 16:31:28.066105 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.066069    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-node-exporter-wtmp\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.066282 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.066123    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmzl5\" (UniqueName: \"kubernetes.io/projected/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-kube-api-access-tmzl5\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.066282 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.066164    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-node-exporter-tls\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.066282 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.066191    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.066282 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.066218    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-metrics-client-ca\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.066282 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.066245    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-sys\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.066282 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.066262    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-root\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.066530 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.066306    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-node-exporter-textfile\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.066530 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.066330    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.167664 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.167627    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-node-exporter-tls\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.167664 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.167668    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.167929 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.167701    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-metrics-client-ca\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.167929 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.167728    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-sys\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.167929 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.167750    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-root\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.167929 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.167801    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-node-exporter-textfile\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.167929 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.167823    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.167929 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.167885    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-node-exporter-wtmp\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.167929 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.167894    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-sys\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.167929 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.167918    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmzl5\" (UniqueName: \"kubernetes.io/projected/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-kube-api-access-tmzl5\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.168290 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.168092    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-node-exporter-wtmp\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.168290 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.168137    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-root\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.168402 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.168369    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.168402 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.168392    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-node-exporter-textfile\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.168698 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.168670    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-metrics-client-ca\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.170341 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.170318    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.170502 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.170487    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-node-exporter-tls\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.176661 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.176638    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmzl5\" (UniqueName: \"kubernetes.io/projected/2973e6a7-3ef6-4f3e-9233-e0435d98a6e2-kube-api-access-tmzl5\") pod \"node-exporter-2wnvq\" (UID: \"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2\") " pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.286510 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.286421    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2wnvq"
Sep 08 16:31:28.587008 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:31:28.586984    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2973e6a7_3ef6_4f3e_9233_e0435d98a6e2.slice/crio-2bfb9f971ec6c30de1ecf1cf1b7250a1ec4f7d7cb06d8327320793221fd2e332 WatchSource:0}: Error finding container 2bfb9f971ec6c30de1ecf1cf1b7250a1ec4f7d7cb06d8327320793221fd2e332: Status 404 returned error can't find the container with id 2bfb9f971ec6c30de1ecf1cf1b7250a1ec4f7d7cb06d8327320793221fd2e332
Sep 08 16:31:28.687284 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:28.687256    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2wnvq" event={"ID":"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2","Type":"ContainerStarted","Data":"2bfb9f971ec6c30de1ecf1cf1b7250a1ec4f7d7cb06d8327320793221fd2e332"}
Sep 08 16:31:29.692930 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:29.692888    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mz7tj" event={"ID":"50d7b024-55d7-4077-8584-9657bf3008b5","Type":"ContainerStarted","Data":"22c64f7c499747e8864a29bd618a22992b9edfabe4aa9d1ba6d3593e13433420"}
Sep 08 16:31:29.693383 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:29.692952    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:31:29.694418 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:29.694397    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g67mm" event={"ID":"0abcdd93-6ae1-4e51-a19b-911c06152822","Type":"ContainerStarted","Data":"f0cc2f7d4506db305fc14e8c5de72c1ed13bfec38476612cfd92c78bcca500c5"}
Sep 08 16:31:29.733519 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:29.733463    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mz7tj" podStartSLOduration=67.680489242 podStartE2EDuration="1m13.733440826s" totalImagesPullingTime="6.052951584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:31:29.714296048 +0000 UTC m=+73.877267489" watchObservedRunningTime="2026-09-08 16:31:29.733440826 +0000 UTC m=+73.896412274"
Sep 08 16:31:29.733698 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:29.733676    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-g67mm" podStartSLOduration=67.692317075 podStartE2EDuration="1m13.733669371s" totalImagesPullingTime="6.041352296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:31:29.731973575 +0000 UTC m=+73.894945024" watchObservedRunningTime="2026-09-08 16:31:29.733669371 +0000 UTC m=+73.896640817"
Sep 08 16:31:30.699504 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:30.699468    2557 generic.go:356] "Generic (PLEG): container finished" podID="2973e6a7-3ef6-4f3e-9233-e0435d98a6e2" containerID="74e300e20afb1b82ad0c75ef3526f714f542d20cde133fd67df973408af606ce" exitCode=0
Sep 08 16:31:30.699955 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:30.699556    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2wnvq" event={"ID":"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2","Type":"ContainerDied","Data":"74e300e20afb1b82ad0c75ef3526f714f542d20cde133fd67df973408af606ce"}
Sep 08 16:31:31.704930 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:31.704894    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2wnvq" event={"ID":"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2","Type":"ContainerStarted","Data":"1aabb2292c337e9df3aac2ed8d5a0d02726102aafd8ef63a25aaf3727425493e"}
Sep 08 16:31:31.705304 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:31.704945    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2wnvq" event={"ID":"2973e6a7-3ef6-4f3e-9233-e0435d98a6e2","Type":"ContainerStarted","Data":"1882bbc8cac7c7b7ef94a19bee872eeb81799e04e1f9f44627603da8601121af"}
Sep 08 16:31:31.739441 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:31.739391    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2wnvq" podStartSLOduration=2.9225012169999998 podStartE2EDuration="4.739373086s" totalImagesPullingTime="1.719160019s" totalInitContainerRuntime="97.71185ms" isStatefulPod=true podCreationTimestamp="2026-09-08 16:31:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:31:31.737788012 +0000 UTC m=+75.900759460" watchObservedRunningTime="2026-09-08 16:31:31.739373086 +0000 UTC m=+75.902344567"
Sep 08 16:31:34.597116 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:34.597083    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:31:36.627403 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:36.627377    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7967f76576-9zbkf"
Sep 08 16:31:39.613018 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:39.612955    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr" podUID="00fce104-5845-4461-8d09-b70ebad8c5b2" containerName="registry" containerID="cri-o://941a761d29cb62e83833a9554a041be57ed7a7b5968beaf211b6c81e03d62d8e" gracePeriod=30
Sep 08 16:31:39.732527 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:39.732501    2557 generic.go:356] "Generic (PLEG): container finished" podID="00fce104-5845-4461-8d09-b70ebad8c5b2" containerID="941a761d29cb62e83833a9554a041be57ed7a7b5968beaf211b6c81e03d62d8e" exitCode=0
Sep 08 16:31:39.732667 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:39.732579    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr" event={"ID":"00fce104-5845-4461-8d09-b70ebad8c5b2","Type":"ContainerDied","Data":"941a761d29cb62e83833a9554a041be57ed7a7b5968beaf211b6c81e03d62d8e"}
Sep 08 16:31:39.850069 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:39.850047    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:31:39.955140 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:39.955102    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qw6sv\" (UniqueName: \"kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-kube-api-access-qw6sv\") pod \"00fce104-5845-4461-8d09-b70ebad8c5b2\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") "
Sep 08 16:31:39.955337 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:39.955163    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-tls\") pod \"00fce104-5845-4461-8d09-b70ebad8c5b2\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") "
Sep 08 16:31:39.955337 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:39.955187    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00fce104-5845-4461-8d09-b70ebad8c5b2-image-registry-private-configuration\") pod \"00fce104-5845-4461-8d09-b70ebad8c5b2\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") "
Sep 08 16:31:39.955337 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:39.955216    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00fce104-5845-4461-8d09-b70ebad8c5b2-trusted-ca\") pod \"00fce104-5845-4461-8d09-b70ebad8c5b2\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") "
Sep 08 16:31:39.955337 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:39.955241    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00fce104-5845-4461-8d09-b70ebad8c5b2-ca-trust-extracted\") pod \"00fce104-5845-4461-8d09-b70ebad8c5b2\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") "
Sep 08 16:31:39.955337 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:39.955259    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-bound-sa-token\") pod \"00fce104-5845-4461-8d09-b70ebad8c5b2\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") "
Sep 08 16:31:39.955337 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:39.955293    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00fce104-5845-4461-8d09-b70ebad8c5b2-installation-pull-secrets\") pod \"00fce104-5845-4461-8d09-b70ebad8c5b2\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") "
Sep 08 16:31:39.955815 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:39.955731    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-certificates\") pod \"00fce104-5845-4461-8d09-b70ebad8c5b2\" (UID: \"00fce104-5845-4461-8d09-b70ebad8c5b2\") "
Sep 08 16:31:39.955815 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:39.955753    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00fce104-5845-4461-8d09-b70ebad8c5b2-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "00fce104-5845-4461-8d09-b70ebad8c5b2" (UID: "00fce104-5845-4461-8d09-b70ebad8c5b2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:31:39.956447 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:39.956412    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "00fce104-5845-4461-8d09-b70ebad8c5b2" (UID: "00fce104-5845-4461-8d09-b70ebad8c5b2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:31:39.956553 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:39.956535    2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00fce104-5845-4461-8d09-b70ebad8c5b2-trusted-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:31:39.957987 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:39.957956    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "00fce104-5845-4461-8d09-b70ebad8c5b2" (UID: "00fce104-5845-4461-8d09-b70ebad8c5b2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:31:39.958091 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:39.958048    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00fce104-5845-4461-8d09-b70ebad8c5b2-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "00fce104-5845-4461-8d09-b70ebad8c5b2" (UID: "00fce104-5845-4461-8d09-b70ebad8c5b2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:31:39.958180 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:39.958154    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00fce104-5845-4461-8d09-b70ebad8c5b2-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "00fce104-5845-4461-8d09-b70ebad8c5b2" (UID: "00fce104-5845-4461-8d09-b70ebad8c5b2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:31:39.958239 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:39.958208    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-kube-api-access-qw6sv" (OuterVolumeSpecName: "kube-api-access-qw6sv") pod "00fce104-5845-4461-8d09-b70ebad8c5b2" (UID: "00fce104-5845-4461-8d09-b70ebad8c5b2"). InnerVolumeSpecName "kube-api-access-qw6sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:31:39.958292 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:39.958243    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "00fce104-5845-4461-8d09-b70ebad8c5b2" (UID: "00fce104-5845-4461-8d09-b70ebad8c5b2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:31:39.964039 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:39.964013    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00fce104-5845-4461-8d09-b70ebad8c5b2-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "00fce104-5845-4461-8d09-b70ebad8c5b2" (UID: "00fce104-5845-4461-8d09-b70ebad8c5b2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:31:40.057796 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:40.057757    2557 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00fce104-5845-4461-8d09-b70ebad8c5b2-ca-trust-extracted\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:31:40.057796 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:40.057789    2557 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-bound-sa-token\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:31:40.057796 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:40.057798    2557 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00fce104-5845-4461-8d09-b70ebad8c5b2-installation-pull-secrets\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:31:40.058038 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:40.057807    2557 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-certificates\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:31:40.058038 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:40.057815    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw6sv\" (UniqueName: \"kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-kube-api-access-qw6sv\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:31:40.058038 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:40.057823    2557 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00fce104-5845-4461-8d09-b70ebad8c5b2-registry-tls\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:31:40.058038 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:40.057850    2557 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00fce104-5845-4461-8d09-b70ebad8c5b2-image-registry-private-configuration\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:31:40.737590 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:40.737565    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"
Sep 08 16:31:40.738114 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:40.737565    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d54d7dd4b-ccnlr" event={"ID":"00fce104-5845-4461-8d09-b70ebad8c5b2","Type":"ContainerDied","Data":"fcaf3b8d14ea8b2b9a6f3f4370a26d0c86fd26d86e11e2529ce5f9ba0b9b5d4e"}
Sep 08 16:31:40.738114 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:40.737708    2557 scope.go:117] "RemoveContainer" containerID="941a761d29cb62e83833a9554a041be57ed7a7b5968beaf211b6c81e03d62d8e"
Sep 08 16:31:40.755036 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:40.755005    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"]
Sep 08 16:31:40.759396 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:40.759373    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7d54d7dd4b-ccnlr"]
Sep 08 16:31:42.344612 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:31:42.344579    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00fce104-5845-4461-8d09-b70ebad8c5b2" path="/var/lib/kubelet/pods/00fce104-5845-4461-8d09-b70ebad8c5b2/volumes"
Sep 08 16:32:00.702751 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:32:00.702721    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mz7tj"
Sep 08 16:35:16.221308 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:35:16.221281    2557 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 16:36:38.789722 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.789679    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf"]
Sep 08 16:36:38.790376 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.790064    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00fce104-5845-4461-8d09-b70ebad8c5b2" containerName="registry"
Sep 08 16:36:38.790376 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.790082    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="00fce104-5845-4461-8d09-b70ebad8c5b2" containerName="registry"
Sep 08 16:36:38.790376 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.790146    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="00fce104-5845-4461-8d09-b70ebad8c5b2" containerName="registry"
Sep 08 16:36:38.792155 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.792134    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf"
Sep 08 16:36:38.795066 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.795038    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 08 16:36:38.795066 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.795056    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 08 16:36:38.795290 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.795156    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dmjj6\""
Sep 08 16:36:38.795342 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.795294    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 08 16:36:38.795442 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.795428    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 08 16:36:38.820263 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.820234    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf"]
Sep 08 16:36:38.865064 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.865032    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/077f72be-6477-4467-b428-fe19839b9934-tmp\") pod \"opendatahub-operator-controller-manager-86d7bb586f-xxbqf\" (UID: \"077f72be-6477-4467-b428-fe19839b9934\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf"
Sep 08 16:36:38.865232 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.865088    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/077f72be-6477-4467-b428-fe19839b9934-manifests\") pod \"opendatahub-operator-controller-manager-86d7bb586f-xxbqf\" (UID: \"077f72be-6477-4467-b428-fe19839b9934\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf"
Sep 08 16:36:38.865232 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.865109    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/077f72be-6477-4467-b428-fe19839b9934-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86d7bb586f-xxbqf\" (UID: \"077f72be-6477-4467-b428-fe19839b9934\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf"
Sep 08 16:36:38.865232 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.865174    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/077f72be-6477-4467-b428-fe19839b9934-webhook-cert\") pod \"opendatahub-operator-controller-manager-86d7bb586f-xxbqf\" (UID: \"077f72be-6477-4467-b428-fe19839b9934\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf"
Sep 08 16:36:38.865232 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.865209    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mjrs\" (UniqueName: \"kubernetes.io/projected/077f72be-6477-4467-b428-fe19839b9934-kube-api-access-5mjrs\") pod \"opendatahub-operator-controller-manager-86d7bb586f-xxbqf\" (UID: \"077f72be-6477-4467-b428-fe19839b9934\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf"
Sep 08 16:36:38.966366 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.966324    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mjrs\" (UniqueName: \"kubernetes.io/projected/077f72be-6477-4467-b428-fe19839b9934-kube-api-access-5mjrs\") pod \"opendatahub-operator-controller-manager-86d7bb586f-xxbqf\" (UID: \"077f72be-6477-4467-b428-fe19839b9934\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf"
Sep 08 16:36:38.966544 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.966380    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/077f72be-6477-4467-b428-fe19839b9934-tmp\") pod \"opendatahub-operator-controller-manager-86d7bb586f-xxbqf\" (UID: \"077f72be-6477-4467-b428-fe19839b9934\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf"
Sep 08 16:36:38.966544 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.966445    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/077f72be-6477-4467-b428-fe19839b9934-manifests\") pod \"opendatahub-operator-controller-manager-86d7bb586f-xxbqf\" (UID: \"077f72be-6477-4467-b428-fe19839b9934\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf"
Sep 08 16:36:38.966544 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.966472    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/077f72be-6477-4467-b428-fe19839b9934-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86d7bb586f-xxbqf\" (UID: \"077f72be-6477-4467-b428-fe19839b9934\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf"
Sep 08 16:36:38.966544 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.966511    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/077f72be-6477-4467-b428-fe19839b9934-webhook-cert\") pod \"opendatahub-operator-controller-manager-86d7bb586f-xxbqf\" (UID: \"077f72be-6477-4467-b428-fe19839b9934\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf"
Sep 08 16:36:38.966953 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.966930    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/077f72be-6477-4467-b428-fe19839b9934-manifests\") pod \"opendatahub-operator-controller-manager-86d7bb586f-xxbqf\" (UID: \"077f72be-6477-4467-b428-fe19839b9934\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf"
Sep 08 16:36:38.968942 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.968912    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/077f72be-6477-4467-b428-fe19839b9934-tmp\") pod \"opendatahub-operator-controller-manager-86d7bb586f-xxbqf\" (UID: \"077f72be-6477-4467-b428-fe19839b9934\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf"
Sep 08 16:36:38.969176 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.969159    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/077f72be-6477-4467-b428-fe19839b9934-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86d7bb586f-xxbqf\" (UID: \"077f72be-6477-4467-b428-fe19839b9934\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf"
Sep 08 16:36:38.969221 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.969199    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/077f72be-6477-4467-b428-fe19839b9934-webhook-cert\") pod \"opendatahub-operator-controller-manager-86d7bb586f-xxbqf\" (UID: \"077f72be-6477-4467-b428-fe19839b9934\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf"
Sep 08 16:36:38.974667 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:38.974641    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mjrs\" (UniqueName: \"kubernetes.io/projected/077f72be-6477-4467-b428-fe19839b9934-kube-api-access-5mjrs\") pod \"opendatahub-operator-controller-manager-86d7bb586f-xxbqf\" (UID: \"077f72be-6477-4467-b428-fe19839b9934\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf"
Sep 08 16:36:39.102291 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:39.102180    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf"
Sep 08 16:36:39.373486 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:39.373463    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf"]
Sep 08 16:36:39.376334 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:36:39.376308    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077f72be_6477_4467_b428_fe19839b9934.slice/crio-8200940b460d04587b22acd2bb14216bcdd87052a1f9d68a3cb8ac44b19d5749 WatchSource:0}: Error finding container 8200940b460d04587b22acd2bb14216bcdd87052a1f9d68a3cb8ac44b19d5749: Status 404 returned error can't find the container with id 8200940b460d04587b22acd2bb14216bcdd87052a1f9d68a3cb8ac44b19d5749
Sep 08 16:36:39.379001 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:39.378983    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 16:36:39.726385 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:39.726345    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf" event={"ID":"077f72be-6477-4467-b428-fe19839b9934","Type":"ContainerStarted","Data":"8200940b460d04587b22acd2bb14216bcdd87052a1f9d68a3cb8ac44b19d5749"}
Sep 08 16:36:42.740490 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:42.740454    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf" event={"ID":"077f72be-6477-4467-b428-fe19839b9934","Type":"ContainerStarted","Data":"2312471ada46b7b921445465f0b73b0b7babb21764c944be67280721d0f72a25"}
Sep 08 16:36:43.745667 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:43.745633    2557 generic.go:356] "Generic (PLEG): container finished" podID="077f72be-6477-4467-b428-fe19839b9934" containerID="2312471ada46b7b921445465f0b73b0b7babb21764c944be67280721d0f72a25" exitCode=0
Sep 08 16:36:43.746052 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:43.745723    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf" event={"ID":"077f72be-6477-4467-b428-fe19839b9934","Type":"ContainerDied","Data":"2312471ada46b7b921445465f0b73b0b7babb21764c944be67280721d0f72a25"}
Sep 08 16:36:44.751468 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:44.751430    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf" event={"ID":"077f72be-6477-4467-b428-fe19839b9934","Type":"ContainerStarted","Data":"1bd8f874051f79faa33019093bb94462530aade17290f26a6b376f2a2fb1bc96"}
Sep 08 16:36:44.751811 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:44.751532    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf"
Sep 08 16:36:44.802541 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:44.802440    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf" podStartSLOduration=2.078590589 podStartE2EDuration="6.80241861s" totalImagesPullingTime="3.670576175s" totalInitContainerRuntime="1.053251846s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:36:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:36:44.800501534 +0000 UTC m=+388.963472986" watchObservedRunningTime="2026-09-08 16:36:44.80241861 +0000 UTC m=+388.965390057"
Sep 08 16:36:55.757383 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:55.757355    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-86d7bb586f-xxbqf"
Sep 08 16:36:58.644972 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:58.644939    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7469867b85-9j7nx"]
Sep 08 16:36:58.655177 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:58.655149    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7469867b85-9j7nx"
Sep 08 16:36:58.664360 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:58.664332    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cmz69\""
Sep 08 16:36:58.664615 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:58.664599    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 08 16:36:58.665269 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:58.665253    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 08 16:36:58.665351 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:58.665283    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 16:36:58.665351 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:58.665334    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 16:36:58.669029 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:58.669008    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7469867b85-9j7nx"]
Sep 08 16:36:58.720368 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:58.720338    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/614bb2ee-e0c3-4ddb-8034-0a2231ce4608-tls-certs\") pod \"kube-auth-proxy-7469867b85-9j7nx\" (UID: \"614bb2ee-e0c3-4ddb-8034-0a2231ce4608\") " pod="openshift-ingress/kube-auth-proxy-7469867b85-9j7nx"
Sep 08 16:36:58.720551 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:58.720388    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg279\" (UniqueName: \"kubernetes.io/projected/614bb2ee-e0c3-4ddb-8034-0a2231ce4608-kube-api-access-zg279\") pod \"kube-auth-proxy-7469867b85-9j7nx\" (UID: \"614bb2ee-e0c3-4ddb-8034-0a2231ce4608\") " pod="openshift-ingress/kube-auth-proxy-7469867b85-9j7nx"
Sep 08 16:36:58.720551 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:58.720424    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/614bb2ee-e0c3-4ddb-8034-0a2231ce4608-tmp\") pod \"kube-auth-proxy-7469867b85-9j7nx\" (UID: \"614bb2ee-e0c3-4ddb-8034-0a2231ce4608\") " pod="openshift-ingress/kube-auth-proxy-7469867b85-9j7nx"
Sep 08 16:36:58.821568 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:58.821530    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg279\" (UniqueName: \"kubernetes.io/projected/614bb2ee-e0c3-4ddb-8034-0a2231ce4608-kube-api-access-zg279\") pod \"kube-auth-proxy-7469867b85-9j7nx\" (UID: \"614bb2ee-e0c3-4ddb-8034-0a2231ce4608\") " pod="openshift-ingress/kube-auth-proxy-7469867b85-9j7nx"
Sep 08 16:36:58.821749 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:58.821582    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/614bb2ee-e0c3-4ddb-8034-0a2231ce4608-tmp\") pod \"kube-auth-proxy-7469867b85-9j7nx\" (UID: \"614bb2ee-e0c3-4ddb-8034-0a2231ce4608\") " pod="openshift-ingress/kube-auth-proxy-7469867b85-9j7nx"
Sep 08 16:36:58.821749 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:58.821633    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/614bb2ee-e0c3-4ddb-8034-0a2231ce4608-tls-certs\") pod \"kube-auth-proxy-7469867b85-9j7nx\" (UID: \"614bb2ee-e0c3-4ddb-8034-0a2231ce4608\") " pod="openshift-ingress/kube-auth-proxy-7469867b85-9j7nx"
Sep 08 16:36:58.824144 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:58.824115    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/614bb2ee-e0c3-4ddb-8034-0a2231ce4608-tmp\") pod \"kube-auth-proxy-7469867b85-9j7nx\" (UID: \"614bb2ee-e0c3-4ddb-8034-0a2231ce4608\") " pod="openshift-ingress/kube-auth-proxy-7469867b85-9j7nx"
Sep 08 16:36:58.824262 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:58.824246    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/614bb2ee-e0c3-4ddb-8034-0a2231ce4608-tls-certs\") pod \"kube-auth-proxy-7469867b85-9j7nx\" (UID: \"614bb2ee-e0c3-4ddb-8034-0a2231ce4608\") " pod="openshift-ingress/kube-auth-proxy-7469867b85-9j7nx"
Sep 08 16:36:58.830283 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:58.830258    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg279\" (UniqueName: \"kubernetes.io/projected/614bb2ee-e0c3-4ddb-8034-0a2231ce4608-kube-api-access-zg279\") pod \"kube-auth-proxy-7469867b85-9j7nx\" (UID: \"614bb2ee-e0c3-4ddb-8034-0a2231ce4608\") " pod="openshift-ingress/kube-auth-proxy-7469867b85-9j7nx"
Sep 08 16:36:58.963841 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:58.963806    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7469867b85-9j7nx"
Sep 08 16:36:59.237566 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:59.237485    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7469867b85-9j7nx"]
Sep 08 16:36:59.240507 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:36:59.240480    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod614bb2ee_e0c3_4ddb_8034_0a2231ce4608.slice/crio-8550296679c3f081bf2647c42ce4b4b023413b75a190e0f8d97bd6cce909d384 WatchSource:0}: Error finding container 8550296679c3f081bf2647c42ce4b4b023413b75a190e0f8d97bd6cce909d384: Status 404 returned error can't find the container with id 8550296679c3f081bf2647c42ce4b4b023413b75a190e0f8d97bd6cce909d384
Sep 08 16:36:59.806892 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:59.806855    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7469867b85-9j7nx" event={"ID":"614bb2ee-e0c3-4ddb-8034-0a2231ce4608","Type":"ContainerStarted","Data":"8550296679c3f081bf2647c42ce4b4b023413b75a190e0f8d97bd6cce909d384"}
Sep 08 16:36:59.971176 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:59.971125    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"]
Sep 08 16:36:59.973622 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:59.973586    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:36:59.976290 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:59.976263    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 08 16:36:59.976448 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:59.976379    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xgf4w\""
Sep 08 16:36:59.976546 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:59.976462    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 08 16:36:59.976610 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:59.976560    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 08 16:36:59.976709 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:59.976694    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 08 16:36:59.988649 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:36:59.988625    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"]
Sep 08 16:37:00.031469 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:00.031428    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lrrg\" (UniqueName: \"kubernetes.io/projected/b44c981c-52d1-45b3-be46-f0f876192a7c-kube-api-access-4lrrg\") pod \"istiod-openshift-gateway-7b7995d5cd-g767q\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:00.031671 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:00.031493    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44c981c-52d1-45b3-be46-f0f876192a7c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-g767q\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:00.031671 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:00.031578    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b44c981c-52d1-45b3-be46-f0f876192a7c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-g767q\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:00.031671 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:00.031625    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b44c981c-52d1-45b3-be46-f0f876192a7c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-g767q\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:00.031671 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:00.031657    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b44c981c-52d1-45b3-be46-f0f876192a7c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-g767q\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:00.031952 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:00.031720    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b44c981c-52d1-45b3-be46-f0f876192a7c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-g767q\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:00.031952 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:00.031763    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44c981c-52d1-45b3-be46-f0f876192a7c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-g767q\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:00.132401 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:00.132300    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b44c981c-52d1-45b3-be46-f0f876192a7c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-g767q\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:00.132401 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:00.132353    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b44c981c-52d1-45b3-be46-f0f876192a7c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-g767q\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:00.132401 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:00.132387    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b44c981c-52d1-45b3-be46-f0f876192a7c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-g767q\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:00.132664 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:00.132418    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44c981c-52d1-45b3-be46-f0f876192a7c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-g767q\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:00.132664 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:00.132451    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lrrg\" (UniqueName: \"kubernetes.io/projected/b44c981c-52d1-45b3-be46-f0f876192a7c-kube-api-access-4lrrg\") pod \"istiod-openshift-gateway-7b7995d5cd-g767q\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:00.132664 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:00.132483    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44c981c-52d1-45b3-be46-f0f876192a7c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-g767q\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:00.132664 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:00.132533    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b44c981c-52d1-45b3-be46-f0f876192a7c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-g767q\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:00.133398 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:00.133312    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b44c981c-52d1-45b3-be46-f0f876192a7c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-g767q\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:00.135557 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:00.135509    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b44c981c-52d1-45b3-be46-f0f876192a7c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-g767q\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:00.135681 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:00.135597    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b44c981c-52d1-45b3-be46-f0f876192a7c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-g767q\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:00.135745 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:00.135720    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b44c981c-52d1-45b3-be46-f0f876192a7c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-g767q\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:00.135803 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:00.135757    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44c981c-52d1-45b3-be46-f0f876192a7c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-g767q\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:00.140958 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:00.140929    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44c981c-52d1-45b3-be46-f0f876192a7c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-g767q\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:00.141801 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:00.141773    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lrrg\" (UniqueName: \"kubernetes.io/projected/b44c981c-52d1-45b3-be46-f0f876192a7c-kube-api-access-4lrrg\") pod \"istiod-openshift-gateway-7b7995d5cd-g767q\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:00.286722 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:00.286685    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:00.580028 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:00.579999    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"]
Sep 08 16:37:00.582887 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:37:00.582855    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44c981c_52d1_45b3_be46_f0f876192a7c.slice/crio-1f17fa70119410ff4cda4c68bdf4b2393afb57881a35fd78495c288de10abe22 WatchSource:0}: Error finding container 1f17fa70119410ff4cda4c68bdf4b2393afb57881a35fd78495c288de10abe22: Status 404 returned error can't find the container with id 1f17fa70119410ff4cda4c68bdf4b2393afb57881a35fd78495c288de10abe22
Sep 08 16:37:00.813410 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:00.813364    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q" event={"ID":"b44c981c-52d1-45b3-be46-f0f876192a7c","Type":"ContainerStarted","Data":"1f17fa70119410ff4cda4c68bdf4b2393afb57881a35fd78495c288de10abe22"}
Sep 08 16:37:02.823659 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:02.823614    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7469867b85-9j7nx" event={"ID":"614bb2ee-e0c3-4ddb-8034-0a2231ce4608","Type":"ContainerStarted","Data":"517933d7c2f369484761d6807a2ed3b864f1e3788d9723ae6850477afd6150ab"}
Sep 08 16:37:02.842394 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:02.842337    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7469867b85-9j7nx" podStartSLOduration=1.93648558 podStartE2EDuration="4.842313072s" totalImagesPullingTime="2.905827492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:36:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:37:02.839615509 +0000 UTC m=+407.002586957" watchObservedRunningTime="2026-09-08 16:37:02.842313072 +0000 UTC m=+407.005284570"
Sep 08 16:37:03.960004 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:03.959968    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:37:03.960391 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:03.960049    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 16:37:04.833822 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:04.833775    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q" event={"ID":"b44c981c-52d1-45b3-be46-f0f876192a7c","Type":"ContainerStarted","Data":"67c368735092fa082acf0bf6810be482a6ded22ed2cb0aedcda2c8b83e66c14f"}
Sep 08 16:37:04.834039 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:04.833875    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:04.854785 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:04.854721    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q" podStartSLOduration=2.480434641 podStartE2EDuration="5.854704195s" totalImagesPullingTime="3.374269554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:37:04.853116155 +0000 UTC m=+409.016087615" watchObservedRunningTime="2026-09-08 16:37:04.854704195 +0000 UTC m=+409.017675643"
Sep 08 16:37:05.839606 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:05.839580    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:18.660202 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:18.660170    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-6hf2m"]
Sep 08 16:37:18.664788 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:18.664765    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6hf2m"
Sep 08 16:37:18.667854 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:18.667814    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-mbzhb\""
Sep 08 16:37:18.668087 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:18.668044    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 08 16:37:18.672591 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:18.672495    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-m8wh8"]
Sep 08 16:37:18.675916 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:18.675893    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m8wh8"
Sep 08 16:37:18.677470 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:18.677440    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-6hf2m"]
Sep 08 16:37:18.678586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:18.678562    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-lpd9c\""
Sep 08 16:37:18.678816 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:18.678795    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 08 16:37:18.689693 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:18.689653    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-m8wh8"]
Sep 08 16:37:18.790759 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:18.790726    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f33f305-953d-4f65-95e8-7858a3f99a9d-cert\") pod \"kserve-controller-manager-d6f4b65b8-6hf2m\" (UID: \"5f33f305-953d-4f65-95e8-7858a3f99a9d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6hf2m"
Sep 08 16:37:18.790759 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:18.790764    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pltd\" (UniqueName: \"kubernetes.io/projected/5f33f305-953d-4f65-95e8-7858a3f99a9d-kube-api-access-4pltd\") pod \"kserve-controller-manager-d6f4b65b8-6hf2m\" (UID: \"5f33f305-953d-4f65-95e8-7858a3f99a9d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6hf2m"
Sep 08 16:37:18.791006 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:18.790790    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqd6j\" (UniqueName: \"kubernetes.io/projected/85e2985e-761b-49a5-a7e4-908881c3690b-kube-api-access-rqd6j\") pod \"llmisvc-controller-manager-777c6bd7d5-m8wh8\" (UID: \"85e2985e-761b-49a5-a7e4-908881c3690b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m8wh8"
Sep 08 16:37:18.791006 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:18.790899    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85e2985e-761b-49a5-a7e4-908881c3690b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-m8wh8\" (UID: \"85e2985e-761b-49a5-a7e4-908881c3690b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m8wh8"
Sep 08 16:37:18.891298 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:18.891261    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85e2985e-761b-49a5-a7e4-908881c3690b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-m8wh8\" (UID: \"85e2985e-761b-49a5-a7e4-908881c3690b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m8wh8"
Sep 08 16:37:18.891298 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:18.891302    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f33f305-953d-4f65-95e8-7858a3f99a9d-cert\") pod \"kserve-controller-manager-d6f4b65b8-6hf2m\" (UID: \"5f33f305-953d-4f65-95e8-7858a3f99a9d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6hf2m"
Sep 08 16:37:18.891515 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:18.891323    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pltd\" (UniqueName: \"kubernetes.io/projected/5f33f305-953d-4f65-95e8-7858a3f99a9d-kube-api-access-4pltd\") pod \"kserve-controller-manager-d6f4b65b8-6hf2m\" (UID: \"5f33f305-953d-4f65-95e8-7858a3f99a9d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6hf2m"
Sep 08 16:37:18.891515 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:18.891351    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqd6j\" (UniqueName: \"kubernetes.io/projected/85e2985e-761b-49a5-a7e4-908881c3690b-kube-api-access-rqd6j\") pod \"llmisvc-controller-manager-777c6bd7d5-m8wh8\" (UID: \"85e2985e-761b-49a5-a7e4-908881c3690b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m8wh8"
Sep 08 16:37:18.891515 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:37:18.891446    2557 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 08 16:37:18.891627 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:37:18.891526    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f33f305-953d-4f65-95e8-7858a3f99a9d-cert podName:5f33f305-953d-4f65-95e8-7858a3f99a9d nodeName:}" failed. No retries permitted until 2026-09-08 16:37:19.391503759 +0000 UTC m=+423.554475200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5f33f305-953d-4f65-95e8-7858a3f99a9d-cert") pod "kserve-controller-manager-d6f4b65b8-6hf2m" (UID: "5f33f305-953d-4f65-95e8-7858a3f99a9d") : secret "kserve-webhook-server-cert" not found
Sep 08 16:37:18.894057 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:18.894029    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85e2985e-761b-49a5-a7e4-908881c3690b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-m8wh8\" (UID: \"85e2985e-761b-49a5-a7e4-908881c3690b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m8wh8"
Sep 08 16:37:18.903728 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:18.903694    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqd6j\" (UniqueName: \"kubernetes.io/projected/85e2985e-761b-49a5-a7e4-908881c3690b-kube-api-access-rqd6j\") pod \"llmisvc-controller-manager-777c6bd7d5-m8wh8\" (UID: \"85e2985e-761b-49a5-a7e4-908881c3690b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m8wh8"
Sep 08 16:37:18.910693 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:18.910631    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pltd\" (UniqueName: \"kubernetes.io/projected/5f33f305-953d-4f65-95e8-7858a3f99a9d-kube-api-access-4pltd\") pod \"kserve-controller-manager-d6f4b65b8-6hf2m\" (UID: \"5f33f305-953d-4f65-95e8-7858a3f99a9d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6hf2m"
Sep 08 16:37:18.989950 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:18.989914    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m8wh8"
Sep 08 16:37:19.262179 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:19.262155    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-m8wh8"]
Sep 08 16:37:19.263591 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:37:19.263561    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod85e2985e_761b_49a5_a7e4_908881c3690b.slice/crio-1213f206ac7f010254ace4f4a0f17eb6d5c1fbc0dba5b687797104d780ccb37f WatchSource:0}: Error finding container 1213f206ac7f010254ace4f4a0f17eb6d5c1fbc0dba5b687797104d780ccb37f: Status 404 returned error can't find the container with id 1213f206ac7f010254ace4f4a0f17eb6d5c1fbc0dba5b687797104d780ccb37f
Sep 08 16:37:19.395635 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:19.395604    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f33f305-953d-4f65-95e8-7858a3f99a9d-cert\") pod \"kserve-controller-manager-d6f4b65b8-6hf2m\" (UID: \"5f33f305-953d-4f65-95e8-7858a3f99a9d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6hf2m"
Sep 08 16:37:19.398116 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:19.398097    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f33f305-953d-4f65-95e8-7858a3f99a9d-cert\") pod \"kserve-controller-manager-d6f4b65b8-6hf2m\" (UID: \"5f33f305-953d-4f65-95e8-7858a3f99a9d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6hf2m"
Sep 08 16:37:19.578684 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:19.578599    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6hf2m"
Sep 08 16:37:19.606128 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:19.606094    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"]
Sep 08 16:37:19.606375 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:19.606346    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q" podUID="b44c981c-52d1-45b3-be46-f0f876192a7c" containerName="discovery" containerID="cri-o://67c368735092fa082acf0bf6810be482a6ded22ed2cb0aedcda2c8b83e66c14f" gracePeriod=30
Sep 08 16:37:19.848019 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:19.847990    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:19.850137 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:19.850113    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-6hf2m"]
Sep 08 16:37:19.853129 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:37:19.853108    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f33f305_953d_4f65_95e8_7858a3f99a9d.slice/crio-41a558628d434414689ac6db3fce95f43f2dc53afcb606339ff7a72e9723c16d WatchSource:0}: Error finding container 41a558628d434414689ac6db3fce95f43f2dc53afcb606339ff7a72e9723c16d: Status 404 returned error can't find the container with id 41a558628d434414689ac6db3fce95f43f2dc53afcb606339ff7a72e9723c16d
Sep 08 16:37:19.885183 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:19.885151    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m8wh8" event={"ID":"85e2985e-761b-49a5-a7e4-908881c3690b","Type":"ContainerStarted","Data":"1213f206ac7f010254ace4f4a0f17eb6d5c1fbc0dba5b687797104d780ccb37f"}
Sep 08 16:37:19.886538 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:19.886509    2557 generic.go:356] "Generic (PLEG): container finished" podID="b44c981c-52d1-45b3-be46-f0f876192a7c" containerID="67c368735092fa082acf0bf6810be482a6ded22ed2cb0aedcda2c8b83e66c14f" exitCode=0
Sep 08 16:37:19.886703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:19.886564    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"
Sep 08 16:37:19.886703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:19.886599    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q" event={"ID":"b44c981c-52d1-45b3-be46-f0f876192a7c","Type":"ContainerDied","Data":"67c368735092fa082acf0bf6810be482a6ded22ed2cb0aedcda2c8b83e66c14f"}
Sep 08 16:37:19.886703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:19.886645    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q" event={"ID":"b44c981c-52d1-45b3-be46-f0f876192a7c","Type":"ContainerDied","Data":"1f17fa70119410ff4cda4c68bdf4b2393afb57881a35fd78495c288de10abe22"}
Sep 08 16:37:19.886703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:19.886672    2557 scope.go:117] "RemoveContainer" containerID="67c368735092fa082acf0bf6810be482a6ded22ed2cb0aedcda2c8b83e66c14f"
Sep 08 16:37:19.888053 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:19.888031    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6hf2m" event={"ID":"5f33f305-953d-4f65-95e8-7858a3f99a9d","Type":"ContainerStarted","Data":"41a558628d434414689ac6db3fce95f43f2dc53afcb606339ff7a72e9723c16d"}
Sep 08 16:37:19.895441 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:19.895419    2557 scope.go:117] "RemoveContainer" containerID="67c368735092fa082acf0bf6810be482a6ded22ed2cb0aedcda2c8b83e66c14f"
Sep 08 16:37:19.895741 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:37:19.895720    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67c368735092fa082acf0bf6810be482a6ded22ed2cb0aedcda2c8b83e66c14f\": container with ID starting with 67c368735092fa082acf0bf6810be482a6ded22ed2cb0aedcda2c8b83e66c14f not found: ID does not exist" containerID="67c368735092fa082acf0bf6810be482a6ded22ed2cb0aedcda2c8b83e66c14f"
Sep 08 16:37:19.895816 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:19.895754    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67c368735092fa082acf0bf6810be482a6ded22ed2cb0aedcda2c8b83e66c14f"} err="failed to get container status \"67c368735092fa082acf0bf6810be482a6ded22ed2cb0aedcda2c8b83e66c14f\": rpc error: code = NotFound desc = could not find container \"67c368735092fa082acf0bf6810be482a6ded22ed2cb0aedcda2c8b83e66c14f\": container with ID starting with 67c368735092fa082acf0bf6810be482a6ded22ed2cb0aedcda2c8b83e66c14f not found: ID does not exist"
Sep 08 16:37:20.001109 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:20.001073    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44c981c-52d1-45b3-be46-f0f876192a7c-istio-token\") pod \"b44c981c-52d1-45b3-be46-f0f876192a7c\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") "
Sep 08 16:37:20.001285 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:20.001131    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b44c981c-52d1-45b3-be46-f0f876192a7c-cacerts\") pod \"b44c981c-52d1-45b3-be46-f0f876192a7c\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") "
Sep 08 16:37:20.001285 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:20.001159    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b44c981c-52d1-45b3-be46-f0f876192a7c-istio-csr-ca-configmap\") pod \"b44c981c-52d1-45b3-be46-f0f876192a7c\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") "
Sep 08 16:37:20.001285 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:20.001203    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lrrg\" (UniqueName: \"kubernetes.io/projected/b44c981c-52d1-45b3-be46-f0f876192a7c-kube-api-access-4lrrg\") pod \"b44c981c-52d1-45b3-be46-f0f876192a7c\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") "
Sep 08 16:37:20.001285 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:20.001228    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44c981c-52d1-45b3-be46-f0f876192a7c-local-certs\") pod \"b44c981c-52d1-45b3-be46-f0f876192a7c\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") "
Sep 08 16:37:20.001285 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:20.001248    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b44c981c-52d1-45b3-be46-f0f876192a7c-istio-kubeconfig\") pod \"b44c981c-52d1-45b3-be46-f0f876192a7c\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") "
Sep 08 16:37:20.001285 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:20.001271    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b44c981c-52d1-45b3-be46-f0f876192a7c-istio-csr-dns-cert\") pod \"b44c981c-52d1-45b3-be46-f0f876192a7c\" (UID: \"b44c981c-52d1-45b3-be46-f0f876192a7c\") "
Sep 08 16:37:20.001748 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:20.001706    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44c981c-52d1-45b3-be46-f0f876192a7c-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "b44c981c-52d1-45b3-be46-f0f876192a7c" (UID: "b44c981c-52d1-45b3-be46-f0f876192a7c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:37:20.003887 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:20.003848    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b44c981c-52d1-45b3-be46-f0f876192a7c-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "b44c981c-52d1-45b3-be46-f0f876192a7c" (UID: "b44c981c-52d1-45b3-be46-f0f876192a7c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:37:20.004008 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:20.003919    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44c981c-52d1-45b3-be46-f0f876192a7c-kube-api-access-4lrrg" (OuterVolumeSpecName: "kube-api-access-4lrrg") pod "b44c981c-52d1-45b3-be46-f0f876192a7c" (UID: "b44c981c-52d1-45b3-be46-f0f876192a7c"). InnerVolumeSpecName "kube-api-access-4lrrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:37:20.004070 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:20.004017    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b44c981c-52d1-45b3-be46-f0f876192a7c-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "b44c981c-52d1-45b3-be46-f0f876192a7c" (UID: "b44c981c-52d1-45b3-be46-f0f876192a7c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:37:20.004117 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:20.004090    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44c981c-52d1-45b3-be46-f0f876192a7c-local-certs" (OuterVolumeSpecName: "local-certs") pod "b44c981c-52d1-45b3-be46-f0f876192a7c" (UID: "b44c981c-52d1-45b3-be46-f0f876192a7c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:37:20.004167 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:20.004152    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44c981c-52d1-45b3-be46-f0f876192a7c-istio-token" (OuterVolumeSpecName: "istio-token") pod "b44c981c-52d1-45b3-be46-f0f876192a7c" (UID: "b44c981c-52d1-45b3-be46-f0f876192a7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:37:20.004207 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:20.004186    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b44c981c-52d1-45b3-be46-f0f876192a7c-cacerts" (OuterVolumeSpecName: "cacerts") pod "b44c981c-52d1-45b3-be46-f0f876192a7c" (UID: "b44c981c-52d1-45b3-be46-f0f876192a7c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:37:20.102678 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:20.102608    2557 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b44c981c-52d1-45b3-be46-f0f876192a7c-cacerts\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:37:20.102678 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:20.102640    2557 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b44c981c-52d1-45b3-be46-f0f876192a7c-istio-csr-ca-configmap\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:37:20.102678 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:20.102674    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lrrg\" (UniqueName: \"kubernetes.io/projected/b44c981c-52d1-45b3-be46-f0f876192a7c-kube-api-access-4lrrg\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:37:20.102883 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:20.102699    2557 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44c981c-52d1-45b3-be46-f0f876192a7c-local-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:37:20.102883 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:20.102708    2557 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b44c981c-52d1-45b3-be46-f0f876192a7c-istio-kubeconfig\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:37:20.102883 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:20.102715    2557 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b44c981c-52d1-45b3-be46-f0f876192a7c-istio-csr-dns-cert\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:37:20.102883 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:20.102723    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44c981c-52d1-45b3-be46-f0f876192a7c-istio-token\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:37:20.211802 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:20.211773    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"]
Sep 08 16:37:20.219144 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:20.219116    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g767q"]
Sep 08 16:37:20.345271 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:20.345235    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b44c981c-52d1-45b3-be46-f0f876192a7c" path="/var/lib/kubelet/pods/b44c981c-52d1-45b3-be46-f0f876192a7c/volumes"
Sep 08 16:37:22.905136 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:22.905099    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m8wh8" event={"ID":"85e2985e-761b-49a5-a7e4-908881c3690b","Type":"ContainerStarted","Data":"45c23faca28bf4df3b79ab3deab473e9ae805038e270c2ed9f22aa0bab564b3c"}
Sep 08 16:37:22.905564 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:22.905213    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m8wh8"
Sep 08 16:37:22.906734 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:22.906707    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6hf2m" event={"ID":"5f33f305-953d-4f65-95e8-7858a3f99a9d","Type":"ContainerStarted","Data":"893032a3ce8b0a1515f2a83868826967b555da681ddc88f8f743e4d8f367f013"}
Sep 08 16:37:22.906854 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:22.906761    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6hf2m"
Sep 08 16:37:22.923143 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:22.923091    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m8wh8" podStartSLOduration=1.3939909670000001 podStartE2EDuration="4.923074358s" totalImagesPullingTime="3.529083391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:37:22.921478522 +0000 UTC m=+427.084449970" watchObservedRunningTime="2026-09-08 16:37:22.923074358 +0000 UTC m=+427.086045805"
Sep 08 16:37:22.939977 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:22.939929    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6hf2m" podStartSLOduration=1.9988892539999998 podStartE2EDuration="4.939911392s" totalImagesPullingTime="2.941022138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:37:22.939110797 +0000 UTC m=+427.102082246" watchObservedRunningTime="2026-09-08 16:37:22.939911392 +0000 UTC m=+427.102882839"
Sep 08 16:37:53.912026 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:53.911953    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m8wh8"
Sep 08 16:37:53.914954 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:37:53.914932    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6hf2m"
Sep 08 16:38:28.024106 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:28.024070    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nk28s"]
Sep 08 16:38:28.024510 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:28.024356    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44c981c-52d1-45b3-be46-f0f876192a7c" containerName="discovery"
Sep 08 16:38:28.024510 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:28.024368    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44c981c-52d1-45b3-be46-f0f876192a7c" containerName="discovery"
Sep 08 16:38:28.024510 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:28.024423    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b44c981c-52d1-45b3-be46-f0f876192a7c" containerName="discovery"
Sep 08 16:38:28.026943 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:28.026927    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nk28s"
Sep 08 16:38:28.029361 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:28.029323    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 08 16:38:28.029361 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:28.029323    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 08 16:38:28.029550 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:28.029369    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-6hjfp\""
Sep 08 16:38:28.029550 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:28.029480    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 08 16:38:28.035727 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:28.035707    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nk28s"]
Sep 08 16:38:28.116829 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:28.116793    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9lmh\" (UniqueName: \"kubernetes.io/projected/9a32e226-5012-4165-a82f-e3c8991a7dc0-kube-api-access-d9lmh\") pod \"dns-operator-controller-manager-7b84cd7c9c-nk28s\" (UID: \"9a32e226-5012-4165-a82f-e3c8991a7dc0\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nk28s"
Sep 08 16:38:28.217746 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:28.217711    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9lmh\" (UniqueName: \"kubernetes.io/projected/9a32e226-5012-4165-a82f-e3c8991a7dc0-kube-api-access-d9lmh\") pod \"dns-operator-controller-manager-7b84cd7c9c-nk28s\" (UID: \"9a32e226-5012-4165-a82f-e3c8991a7dc0\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nk28s"
Sep 08 16:38:28.225850 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:28.225799    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9lmh\" (UniqueName: \"kubernetes.io/projected/9a32e226-5012-4165-a82f-e3c8991a7dc0-kube-api-access-d9lmh\") pod \"dns-operator-controller-manager-7b84cd7c9c-nk28s\" (UID: \"9a32e226-5012-4165-a82f-e3c8991a7dc0\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nk28s"
Sep 08 16:38:28.337151 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:28.337055    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nk28s"
Sep 08 16:38:28.605978 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:28.605952    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nk28s"]
Sep 08 16:38:28.608540 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:38:28.608502    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a32e226_5012_4165_a82f_e3c8991a7dc0.slice/crio-7a1413e6d2199635938552cdcdef65ad984cda6519f7052e7fc6e2b1723a07c5 WatchSource:0}: Error finding container 7a1413e6d2199635938552cdcdef65ad984cda6519f7052e7fc6e2b1723a07c5: Status 404 returned error can't find the container with id 7a1413e6d2199635938552cdcdef65ad984cda6519f7052e7fc6e2b1723a07c5
Sep 08 16:38:29.138674 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:29.138639    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nk28s" event={"ID":"9a32e226-5012-4165-a82f-e3c8991a7dc0","Type":"ContainerStarted","Data":"7a1413e6d2199635938552cdcdef65ad984cda6519f7052e7fc6e2b1723a07c5"}
Sep 08 16:38:30.118025 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:30.117995    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz"]
Sep 08 16:38:30.120627 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:30.120601    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz"
Sep 08 16:38:30.123377 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:30.123344    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-ttpvj\""
Sep 08 16:38:30.136450 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:30.136425    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz"]
Sep 08 16:38:30.233423 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:30.233383    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcqjr\" (UniqueName: \"kubernetes.io/projected/1b7a7b74-be24-4f77-8a70-a869b01aa5c3-kube-api-access-kcqjr\") pod \"limitador-operator-controller-manager-74495c69f8-9dflz\" (UID: \"1b7a7b74-be24-4f77-8a70-a869b01aa5c3\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz"
Sep 08 16:38:30.333821 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:30.333784    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcqjr\" (UniqueName: \"kubernetes.io/projected/1b7a7b74-be24-4f77-8a70-a869b01aa5c3-kube-api-access-kcqjr\") pod \"limitador-operator-controller-manager-74495c69f8-9dflz\" (UID: \"1b7a7b74-be24-4f77-8a70-a869b01aa5c3\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz"
Sep 08 16:38:30.374846 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:30.374759    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcqjr\" (UniqueName: \"kubernetes.io/projected/1b7a7b74-be24-4f77-8a70-a869b01aa5c3-kube-api-access-kcqjr\") pod \"limitador-operator-controller-manager-74495c69f8-9dflz\" (UID: \"1b7a7b74-be24-4f77-8a70-a869b01aa5c3\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz"
Sep 08 16:38:30.431102 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:30.431072    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz"
Sep 08 16:38:30.706367 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:30.706337    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz"]
Sep 08 16:38:30.709442 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:38:30.709417    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7a7b74_be24_4f77_8a70_a869b01aa5c3.slice/crio-e887bd87febe2217280d077c85588f39ea510eff053597220c55f276f8675068 WatchSource:0}: Error finding container e887bd87febe2217280d077c85588f39ea510eff053597220c55f276f8675068: Status 404 returned error can't find the container with id e887bd87febe2217280d077c85588f39ea510eff053597220c55f276f8675068
Sep 08 16:38:31.146924 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:31.146829    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz" event={"ID":"1b7a7b74-be24-4f77-8a70-a869b01aa5c3","Type":"ContainerStarted","Data":"e887bd87febe2217280d077c85588f39ea510eff053597220c55f276f8675068"}
Sep 08 16:38:33.157651 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:33.157615    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz" event={"ID":"1b7a7b74-be24-4f77-8a70-a869b01aa5c3","Type":"ContainerStarted","Data":"e4ac2c789e76de04a37db427151f34cbae51be5160109ea3dab6de06b2417d00"}
Sep 08 16:38:33.158061 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:33.157728    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz"
Sep 08 16:38:33.196784 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:33.196725    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz" podStartSLOduration=0.90044938 podStartE2EDuration="3.196708313s" totalImagesPullingTime="2.296258933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:38:33.195123789 +0000 UTC m=+497.358095238" watchObservedRunningTime="2026-09-08 16:38:33.196708313 +0000 UTC m=+497.359679758"
Sep 08 16:38:34.704329 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:34.704293    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"]
Sep 08 16:38:34.706341 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:34.706324    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"
Sep 08 16:38:34.709014 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:34.708995    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ts7rd\""
Sep 08 16:38:34.726985 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:34.726962    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"]
Sep 08 16:38:34.872095 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:34.872057    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb486435-63bf-4fba-98b9-2b19050e276b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-jphqp\" (UID: \"bb486435-63bf-4fba-98b9-2b19050e276b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"
Sep 08 16:38:34.872275 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:34.872162    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crrhm\" (UniqueName: \"kubernetes.io/projected/bb486435-63bf-4fba-98b9-2b19050e276b-kube-api-access-crrhm\") pod \"kuadrant-operator-controller-manager-76f8f686bf-jphqp\" (UID: \"bb486435-63bf-4fba-98b9-2b19050e276b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"
Sep 08 16:38:34.973392 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:34.973300    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb486435-63bf-4fba-98b9-2b19050e276b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-jphqp\" (UID: \"bb486435-63bf-4fba-98b9-2b19050e276b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"
Sep 08 16:38:34.973392 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:34.973368    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crrhm\" (UniqueName: \"kubernetes.io/projected/bb486435-63bf-4fba-98b9-2b19050e276b-kube-api-access-crrhm\") pod \"kuadrant-operator-controller-manager-76f8f686bf-jphqp\" (UID: \"bb486435-63bf-4fba-98b9-2b19050e276b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"
Sep 08 16:38:34.973768 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:34.973743    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb486435-63bf-4fba-98b9-2b19050e276b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-jphqp\" (UID: \"bb486435-63bf-4fba-98b9-2b19050e276b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"
Sep 08 16:38:34.982039 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:34.982012    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crrhm\" (UniqueName: \"kubernetes.io/projected/bb486435-63bf-4fba-98b9-2b19050e276b-kube-api-access-crrhm\") pod \"kuadrant-operator-controller-manager-76f8f686bf-jphqp\" (UID: \"bb486435-63bf-4fba-98b9-2b19050e276b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"
Sep 08 16:38:35.015219 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:35.015185    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"
Sep 08 16:38:35.292536 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:35.292509    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"]
Sep 08 16:38:35.294768 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:38:35.294744    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb486435_63bf_4fba_98b9_2b19050e276b.slice/crio-289481600b9b40c378094bf56365e6193faad3bd92db4b8741ad3f499afc51f2 WatchSource:0}: Error finding container 289481600b9b40c378094bf56365e6193faad3bd92db4b8741ad3f499afc51f2: Status 404 returned error can't find the container with id 289481600b9b40c378094bf56365e6193faad3bd92db4b8741ad3f499afc51f2
Sep 08 16:38:36.171272 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:36.171237    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp" event={"ID":"bb486435-63bf-4fba-98b9-2b19050e276b","Type":"ContainerStarted","Data":"289481600b9b40c378094bf56365e6193faad3bd92db4b8741ad3f499afc51f2"}
Sep 08 16:38:37.048678 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:37.048641    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-mrrww"]
Sep 08 16:38:37.050628 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:37.050606    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-mrrww"
Sep 08 16:38:37.053461 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:37.053437    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-t2vkb\""
Sep 08 16:38:37.076041 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:37.076014    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-mrrww"]
Sep 08 16:38:37.088431 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:37.088403    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc7w9\" (UniqueName: \"kubernetes.io/projected/82f2dbf1-fb30-4c46-a95f-892a1a07aa11-kube-api-access-cc7w9\") pod \"authorino-operator-76b478bd78-mrrww\" (UID: \"82f2dbf1-fb30-4c46-a95f-892a1a07aa11\") " pod="kuadrant-system/authorino-operator-76b478bd78-mrrww"
Sep 08 16:38:37.179640 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:37.179601    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nk28s" event={"ID":"9a32e226-5012-4165-a82f-e3c8991a7dc0","Type":"ContainerStarted","Data":"77b31b4a75a81e28f04f1f213553eea26f4dae4c69ef76c27d8d2db59cf7405f"}
Sep 08 16:38:37.180139 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:37.179803    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nk28s"
Sep 08 16:38:37.189068 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:37.189033    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc7w9\" (UniqueName: \"kubernetes.io/projected/82f2dbf1-fb30-4c46-a95f-892a1a07aa11-kube-api-access-cc7w9\") pod \"authorino-operator-76b478bd78-mrrww\" (UID: \"82f2dbf1-fb30-4c46-a95f-892a1a07aa11\") " pod="kuadrant-system/authorino-operator-76b478bd78-mrrww"
Sep 08 16:38:37.199058 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:37.199030    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc7w9\" (UniqueName: \"kubernetes.io/projected/82f2dbf1-fb30-4c46-a95f-892a1a07aa11-kube-api-access-cc7w9\") pod \"authorino-operator-76b478bd78-mrrww\" (UID: \"82f2dbf1-fb30-4c46-a95f-892a1a07aa11\") " pod="kuadrant-system/authorino-operator-76b478bd78-mrrww"
Sep 08 16:38:37.200646 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:37.200602    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nk28s" podStartSLOduration=0.737150706 podStartE2EDuration="9.200585549s" totalImagesPullingTime="8.463434843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:38:37.200119519 +0000 UTC m=+501.363090967" watchObservedRunningTime="2026-09-08 16:38:37.200585549 +0000 UTC m=+501.363556996"
Sep 08 16:38:37.363738 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:37.363655    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-mrrww"
Sep 08 16:38:37.648047 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:37.647966    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-mrrww"]
Sep 08 16:38:37.651621 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:38:37.651598    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f2dbf1_fb30_4c46_a95f_892a1a07aa11.slice/crio-08784560f755b9d52116e9ed7e8265fc341a3dfa2e979d1bbd267bffa65f9641 WatchSource:0}: Error finding container 08784560f755b9d52116e9ed7e8265fc341a3dfa2e979d1bbd267bffa65f9641: Status 404 returned error can't find the container with id 08784560f755b9d52116e9ed7e8265fc341a3dfa2e979d1bbd267bffa65f9641
Sep 08 16:38:38.185878 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:38.185822    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-mrrww" event={"ID":"82f2dbf1-fb30-4c46-a95f-892a1a07aa11","Type":"ContainerStarted","Data":"08784560f755b9d52116e9ed7e8265fc341a3dfa2e979d1bbd267bffa65f9641"}
Sep 08 16:38:40.196562 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:40.196523    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp" event={"ID":"bb486435-63bf-4fba-98b9-2b19050e276b","Type":"ContainerStarted","Data":"7da7819e59e09bad726cb15ed5cac4669a785074d5cffcdb452ec9a0d34d72a3"}
Sep 08 16:38:40.197087 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:40.196662    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"
Sep 08 16:38:40.198627 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:40.198598    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-mrrww" event={"ID":"82f2dbf1-fb30-4c46-a95f-892a1a07aa11","Type":"ContainerStarted","Data":"e9c708401a13652094dc2a00828997b9e67ed4e39ab5b9006d871c3e760d055e"}
Sep 08 16:38:40.198737 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:40.198708    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-mrrww"
Sep 08 16:38:40.220479 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:40.220434    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp" podStartSLOduration=2.123531169 podStartE2EDuration="6.220417996s" totalImagesPullingTime="4.096886827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:38:40.218035579 +0000 UTC m=+504.381007028" watchObservedRunningTime="2026-09-08 16:38:40.220417996 +0000 UTC m=+504.383389444"
Sep 08 16:38:40.237339 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:40.237293    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-mrrww" podStartSLOduration=0.764964022 podStartE2EDuration="3.23727745s" totalImagesPullingTime="2.472313428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:38:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:38:40.235978899 +0000 UTC m=+504.398950359" watchObservedRunningTime="2026-09-08 16:38:40.23727745 +0000 UTC m=+504.400248898"
Sep 08 16:38:42.232100 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:42.232064    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9cnvv"]
Sep 08 16:38:42.234167 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:42.234152    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9cnvv"
Sep 08 16:38:42.237318 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:42.237255    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nbn2c\""
Sep 08 16:38:42.237318 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:42.237295    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 08 16:38:42.237515 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:42.237396    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 08 16:38:42.241822 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:42.241798    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9cnvv"]
Sep 08 16:38:42.330238 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:42.330207    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/87ca0d93-517e-42a7-b46d-eeef810fe708-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9cnvv\" (UID: \"87ca0d93-517e-42a7-b46d-eeef810fe708\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9cnvv"
Sep 08 16:38:42.330423 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:42.330252    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg9rj\" (UniqueName: \"kubernetes.io/projected/87ca0d93-517e-42a7-b46d-eeef810fe708-kube-api-access-fg9rj\") pod \"kuadrant-console-plugin-fbdd4f445-9cnvv\" (UID: \"87ca0d93-517e-42a7-b46d-eeef810fe708\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9cnvv"
Sep 08 16:38:42.330423 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:42.330290    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/87ca0d93-517e-42a7-b46d-eeef810fe708-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9cnvv\" (UID: \"87ca0d93-517e-42a7-b46d-eeef810fe708\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9cnvv"
Sep 08 16:38:42.431273 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:42.431237    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg9rj\" (UniqueName: \"kubernetes.io/projected/87ca0d93-517e-42a7-b46d-eeef810fe708-kube-api-access-fg9rj\") pod \"kuadrant-console-plugin-fbdd4f445-9cnvv\" (UID: \"87ca0d93-517e-42a7-b46d-eeef810fe708\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9cnvv"
Sep 08 16:38:42.431459 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:42.431298    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/87ca0d93-517e-42a7-b46d-eeef810fe708-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9cnvv\" (UID: \"87ca0d93-517e-42a7-b46d-eeef810fe708\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9cnvv"
Sep 08 16:38:42.431459 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:42.431343    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/87ca0d93-517e-42a7-b46d-eeef810fe708-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9cnvv\" (UID: \"87ca0d93-517e-42a7-b46d-eeef810fe708\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9cnvv"
Sep 08 16:38:42.432009 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:42.431984    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/87ca0d93-517e-42a7-b46d-eeef810fe708-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9cnvv\" (UID: \"87ca0d93-517e-42a7-b46d-eeef810fe708\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9cnvv"
Sep 08 16:38:42.433898 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:42.433877    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/87ca0d93-517e-42a7-b46d-eeef810fe708-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9cnvv\" (UID: \"87ca0d93-517e-42a7-b46d-eeef810fe708\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9cnvv"
Sep 08 16:38:42.440819 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:42.440796    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg9rj\" (UniqueName: \"kubernetes.io/projected/87ca0d93-517e-42a7-b46d-eeef810fe708-kube-api-access-fg9rj\") pod \"kuadrant-console-plugin-fbdd4f445-9cnvv\" (UID: \"87ca0d93-517e-42a7-b46d-eeef810fe708\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9cnvv"
Sep 08 16:38:42.543303 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:42.543225    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9cnvv"
Sep 08 16:38:42.812359 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:42.812326    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9cnvv"]
Sep 08 16:38:42.815002 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:38:42.814973    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87ca0d93_517e_42a7_b46d_eeef810fe708.slice/crio-a1347a737bb08e61f0f64487d98fa07ef456588824a66f9d715da1a03fa5bfd5 WatchSource:0}: Error finding container a1347a737bb08e61f0f64487d98fa07ef456588824a66f9d715da1a03fa5bfd5: Status 404 returned error can't find the container with id a1347a737bb08e61f0f64487d98fa07ef456588824a66f9d715da1a03fa5bfd5
Sep 08 16:38:43.212559 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:43.212525    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9cnvv" event={"ID":"87ca0d93-517e-42a7-b46d-eeef810fe708","Type":"ContainerStarted","Data":"a1347a737bb08e61f0f64487d98fa07ef456588824a66f9d715da1a03fa5bfd5"}
Sep 08 16:38:44.165196 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:44.165161    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz"
Sep 08 16:38:48.188743 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:48.188711    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nk28s"
Sep 08 16:38:48.242327 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:48.242288    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9cnvv" event={"ID":"87ca0d93-517e-42a7-b46d-eeef810fe708","Type":"ContainerStarted","Data":"1817e06287af55b9f2ff2546b57c996f898bec73096abff93631558ebdd3691e"}
Sep 08 16:38:48.259648 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:48.259603    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9cnvv" podStartSLOduration=1.566947736 podStartE2EDuration="6.259583951s" totalImagesPullingTime="4.692636215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:38:48.259156746 +0000 UTC m=+512.422128205" watchObservedRunningTime="2026-09-08 16:38:48.259583951 +0000 UTC m=+512.422555398"
Sep 08 16:38:51.205329 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:51.205294    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"
Sep 08 16:38:51.205694 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:51.205363    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-mrrww"
Sep 08 16:38:53.464798 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.464760    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"]
Sep 08 16:38:53.465286 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.464950    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp" podUID="bb486435-63bf-4fba-98b9-2b19050e276b" containerName="manager" containerID="cri-o://7da7819e59e09bad726cb15ed5cac4669a785074d5cffcdb452ec9a0d34d72a3" gracePeriod=2
Sep 08 16:38:53.476334 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.476308    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"]
Sep 08 16:38:53.488672 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:38:53.488631    2557 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"bb486435-63bf-4fba-98b9-2b19050e276b\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-jphqp\": pods \"kuadrant-operator-controller-manager-76f8f686bf-jphqp\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"
Sep 08 16:38:53.491640 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.491607    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zs6rn"]
Sep 08 16:38:53.492253 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.492230    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb486435-63bf-4fba-98b9-2b19050e276b" containerName="manager"
Sep 08 16:38:53.492363 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.492352    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb486435-63bf-4fba-98b9-2b19050e276b" containerName="manager"
Sep 08 16:38:53.492532 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.492512    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb486435-63bf-4fba-98b9-2b19050e276b" containerName="manager"
Sep 08 16:38:53.494545 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.494527    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz"]
Sep 08 16:38:53.495053 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.494707    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz" podUID="1b7a7b74-be24-4f77-8a70-a869b01aa5c3" containerName="manager" containerID="cri-o://e4ac2c789e76de04a37db427151f34cbae51be5160109ea3dab6de06b2417d00" gracePeriod=2
Sep 08 16:38:53.495053 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.494747    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zs6rn"
Sep 08 16:38:53.496989 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:38:53.496957    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jphqp\" is forbidden: User \"system:node:ip-10-0-139-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-51.ec2.internal' and this object" podUID="bb486435-63bf-4fba-98b9-2b19050e276b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"
Sep 08 16:38:53.516407 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.516377    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz"]
Sep 08 16:38:53.517796 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:38:53.517762    2557 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"1b7a7b74-be24-4f77-8a70-a869b01aa5c3\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-9dflz\": pods \"limitador-operator-controller-manager-74495c69f8-9dflz\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz"
Sep 08 16:38:53.520325 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:38:53.520173    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9dflz\" is forbidden: User \"system:node:ip-10-0-139-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-51.ec2.internal' and this object" podUID="1b7a7b74-be24-4f77-8a70-a869b01aa5c3" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz"
Sep 08 16:38:53.521275 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.521253    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zs6rn"]
Sep 08 16:38:53.522535 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.522514    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zs6rn\" (UID: \"d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zs6rn"
Sep 08 16:38:53.522652 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.522565    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc7bc\" (UniqueName: \"kubernetes.io/projected/d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a-kube-api-access-rc7bc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zs6rn\" (UID: \"d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zs6rn"
Sep 08 16:38:53.536195 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:38:53.536143    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jphqp\" is forbidden: User \"system:node:ip-10-0-139-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-51.ec2.internal' and this object" podUID="bb486435-63bf-4fba-98b9-2b19050e276b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"
Sep 08 16:38:53.538013 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.537987    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kp4f4"]
Sep 08 16:38:53.538295 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.538281    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b7a7b74-be24-4f77-8a70-a869b01aa5c3" containerName="manager"
Sep 08 16:38:53.538295 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.538296    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b7a7b74-be24-4f77-8a70-a869b01aa5c3" containerName="manager"
Sep 08 16:38:53.538387 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.538382    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b7a7b74-be24-4f77-8a70-a869b01aa5c3" containerName="manager"
Sep 08 16:38:53.549074 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.549048    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kp4f4"
Sep 08 16:38:53.556768 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.556745    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kp4f4"]
Sep 08 16:38:53.583062 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:38:53.583026    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9dflz\" is forbidden: User \"system:node:ip-10-0-139-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-51.ec2.internal' and this object" podUID="1b7a7b74-be24-4f77-8a70-a869b01aa5c3" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz"
Sep 08 16:38:53.624044 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.623902    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pgv8\" (UniqueName: \"kubernetes.io/projected/97a2dece-cd3b-475c-b3a5-60e042103b7d-kube-api-access-6pgv8\") pod \"limitador-operator-controller-manager-74495c69f8-kp4f4\" (UID: \"97a2dece-cd3b-475c-b3a5-60e042103b7d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kp4f4"
Sep 08 16:38:53.624044 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.623976    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zs6rn\" (UID: \"d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zs6rn"
Sep 08 16:38:53.624369 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.624064    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc7bc\" (UniqueName: \"kubernetes.io/projected/d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a-kube-api-access-rc7bc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zs6rn\" (UID: \"d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zs6rn"
Sep 08 16:38:53.624369 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.624302    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zs6rn\" (UID: \"d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zs6rn"
Sep 08 16:38:53.624929 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:38:53.624895    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jphqp\" is forbidden: User \"system:node:ip-10-0-139-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-51.ec2.internal' and this object" podUID="bb486435-63bf-4fba-98b9-2b19050e276b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"
Sep 08 16:38:53.648573 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.648520    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc7bc\" (UniqueName: \"kubernetes.io/projected/d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a-kube-api-access-rc7bc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zs6rn\" (UID: \"d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zs6rn"
Sep 08 16:38:53.724635 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.724604    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pgv8\" (UniqueName: \"kubernetes.io/projected/97a2dece-cd3b-475c-b3a5-60e042103b7d-kube-api-access-6pgv8\") pod \"limitador-operator-controller-manager-74495c69f8-kp4f4\" (UID: \"97a2dece-cd3b-475c-b3a5-60e042103b7d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kp4f4"
Sep 08 16:38:53.727788 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.727762    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"
Sep 08 16:38:53.730218 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.730201    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz"
Sep 08 16:38:53.730316 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:38:53.730242    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9dflz\" is forbidden: User \"system:node:ip-10-0-139-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-51.ec2.internal' and this object" podUID="1b7a7b74-be24-4f77-8a70-a869b01aa5c3" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz"
Sep 08 16:38:53.732277 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:38:53.732256    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jphqp\" is forbidden: User \"system:node:ip-10-0-139-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-51.ec2.internal' and this object" podUID="bb486435-63bf-4fba-98b9-2b19050e276b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"
Sep 08 16:38:53.734803 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:38:53.734779    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9dflz\" is forbidden: User \"system:node:ip-10-0-139-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-51.ec2.internal' and this object" podUID="1b7a7b74-be24-4f77-8a70-a869b01aa5c3" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz"
Sep 08 16:38:53.736647 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:38:53.736628    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jphqp\" is forbidden: User \"system:node:ip-10-0-139-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-51.ec2.internal' and this object" podUID="bb486435-63bf-4fba-98b9-2b19050e276b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"
Sep 08 16:38:53.739415 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.739396    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pgv8\" (UniqueName: \"kubernetes.io/projected/97a2dece-cd3b-475c-b3a5-60e042103b7d-kube-api-access-6pgv8\") pod \"limitador-operator-controller-manager-74495c69f8-kp4f4\" (UID: \"97a2dece-cd3b-475c-b3a5-60e042103b7d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kp4f4"
Sep 08 16:38:53.825674 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.825639    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb486435-63bf-4fba-98b9-2b19050e276b-extensions-socket-volume\") pod \"bb486435-63bf-4fba-98b9-2b19050e276b\" (UID: \"bb486435-63bf-4fba-98b9-2b19050e276b\") "
Sep 08 16:38:53.825782 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.825692    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcqjr\" (UniqueName: \"kubernetes.io/projected/1b7a7b74-be24-4f77-8a70-a869b01aa5c3-kube-api-access-kcqjr\") pod \"1b7a7b74-be24-4f77-8a70-a869b01aa5c3\" (UID: \"1b7a7b74-be24-4f77-8a70-a869b01aa5c3\") "
Sep 08 16:38:53.825782 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.825718    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crrhm\" (UniqueName: \"kubernetes.io/projected/bb486435-63bf-4fba-98b9-2b19050e276b-kube-api-access-crrhm\") pod \"bb486435-63bf-4fba-98b9-2b19050e276b\" (UID: \"bb486435-63bf-4fba-98b9-2b19050e276b\") "
Sep 08 16:38:53.826224 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.826194    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb486435-63bf-4fba-98b9-2b19050e276b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "bb486435-63bf-4fba-98b9-2b19050e276b" (UID: "bb486435-63bf-4fba-98b9-2b19050e276b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:38:53.827992 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.827970    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7a7b74-be24-4f77-8a70-a869b01aa5c3-kube-api-access-kcqjr" (OuterVolumeSpecName: "kube-api-access-kcqjr") pod "1b7a7b74-be24-4f77-8a70-a869b01aa5c3" (UID: "1b7a7b74-be24-4f77-8a70-a869b01aa5c3"). InnerVolumeSpecName "kube-api-access-kcqjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:38:53.828067 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.828037    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb486435-63bf-4fba-98b9-2b19050e276b-kube-api-access-crrhm" (OuterVolumeSpecName: "kube-api-access-crrhm") pod "bb486435-63bf-4fba-98b9-2b19050e276b" (UID: "bb486435-63bf-4fba-98b9-2b19050e276b"). InnerVolumeSpecName "kube-api-access-crrhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:38:53.905439 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.905401    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zs6rn"
Sep 08 16:38:53.912234 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.912202    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kp4f4"
Sep 08 16:38:53.927215 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.927180    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcqjr\" (UniqueName: \"kubernetes.io/projected/1b7a7b74-be24-4f77-8a70-a869b01aa5c3-kube-api-access-kcqjr\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:38:53.927327 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.927221    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crrhm\" (UniqueName: \"kubernetes.io/projected/bb486435-63bf-4fba-98b9-2b19050e276b-kube-api-access-crrhm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:38:53.927327 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:53.927240    2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb486435-63bf-4fba-98b9-2b19050e276b-extensions-socket-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:38:54.194968 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:54.194858    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kp4f4"]
Sep 08 16:38:54.197662 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:54.197639    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zs6rn"]
Sep 08 16:38:54.198118 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:38:54.198096    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a2dece_cd3b_475c_b3a5_60e042103b7d.slice/crio-5f5840ff3ca968359f4e745d3de13ee4fef0562e562cc617e1ee2a72eb6f549a WatchSource:0}: Error finding container 5f5840ff3ca968359f4e745d3de13ee4fef0562e562cc617e1ee2a72eb6f549a: Status 404 returned error can't find the container with id 5f5840ff3ca968359f4e745d3de13ee4fef0562e562cc617e1ee2a72eb6f549a
Sep 08 16:38:54.200020 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:38:54.199995    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b9c1b6_4767_4b47_ba95_77cdb2ef0c3a.slice/crio-093ad35dee9eda4f34bd473200a0cfe7a10a6b607945feda2fb4a87221e00000 WatchSource:0}: Error finding container 093ad35dee9eda4f34bd473200a0cfe7a10a6b607945feda2fb4a87221e00000: Status 404 returned error can't find the container with id 093ad35dee9eda4f34bd473200a0cfe7a10a6b607945feda2fb4a87221e00000
Sep 08 16:38:54.269510 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:54.269481    2557 generic.go:356] "Generic (PLEG): container finished" podID="1b7a7b74-be24-4f77-8a70-a869b01aa5c3" containerID="e4ac2c789e76de04a37db427151f34cbae51be5160109ea3dab6de06b2417d00" exitCode=0
Sep 08 16:38:54.269659 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:54.269535    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz"
Sep 08 16:38:54.269659 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:54.269572    2557 scope.go:117] "RemoveContainer" containerID="e4ac2c789e76de04a37db427151f34cbae51be5160109ea3dab6de06b2417d00"
Sep 08 16:38:54.271439 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:54.271413    2557 generic.go:356] "Generic (PLEG): container finished" podID="bb486435-63bf-4fba-98b9-2b19050e276b" containerID="7da7819e59e09bad726cb15ed5cac4669a785074d5cffcdb452ec9a0d34d72a3" exitCode=0
Sep 08 16:38:54.271558 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:54.271472    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"
Sep 08 16:38:54.271878 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:38:54.271816    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jphqp\" is forbidden: User \"system:node:ip-10-0-139-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-51.ec2.internal' and this object" podUID="bb486435-63bf-4fba-98b9-2b19050e276b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"
Sep 08 16:38:54.274473 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:38:54.274443    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9dflz\" is forbidden: User \"system:node:ip-10-0-139-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-51.ec2.internal' and this object" podUID="1b7a7b74-be24-4f77-8a70-a869b01aa5c3" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz"
Sep 08 16:38:54.276279 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:54.276248    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zs6rn" event={"ID":"d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a","Type":"ContainerStarted","Data":"c63bb8c5d813c498b4888b79c497a28426c6f4b4ce8ebda9077a92a13d6ea432"}
Sep 08 16:38:54.276388 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:54.276303    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zs6rn" event={"ID":"d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a","Type":"ContainerStarted","Data":"093ad35dee9eda4f34bd473200a0cfe7a10a6b607945feda2fb4a87221e00000"}
Sep 08 16:38:54.276764 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:38:54.276623    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9dflz\" is forbidden: User \"system:node:ip-10-0-139-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-51.ec2.internal' and this object" podUID="1b7a7b74-be24-4f77-8a70-a869b01aa5c3" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz"
Sep 08 16:38:54.276893 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:54.276799    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zs6rn"
Sep 08 16:38:54.278673 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:38:54.278484    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jphqp\" is forbidden: User \"system:node:ip-10-0-139-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-51.ec2.internal' and this object" podUID="bb486435-63bf-4fba-98b9-2b19050e276b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"
Sep 08 16:38:54.279462 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:54.279030    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kp4f4" event={"ID":"97a2dece-cd3b-475c-b3a5-60e042103b7d","Type":"ContainerStarted","Data":"3c9cb1cf5a6b613765ddace8478bf8e79a6cd39c57823cd7abd019ebc53f3991"}
Sep 08 16:38:54.279462 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:54.279097    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kp4f4" event={"ID":"97a2dece-cd3b-475c-b3a5-60e042103b7d","Type":"ContainerStarted","Data":"5f5840ff3ca968359f4e745d3de13ee4fef0562e562cc617e1ee2a72eb6f549a"}
Sep 08 16:38:54.279462 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:54.279139    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kp4f4"
Sep 08 16:38:54.279744 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:54.279667    2557 scope.go:117] "RemoveContainer" containerID="e4ac2c789e76de04a37db427151f34cbae51be5160109ea3dab6de06b2417d00"
Sep 08 16:38:54.280013 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:38:54.279990    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4ac2c789e76de04a37db427151f34cbae51be5160109ea3dab6de06b2417d00\": container with ID starting with e4ac2c789e76de04a37db427151f34cbae51be5160109ea3dab6de06b2417d00 not found: ID does not exist" containerID="e4ac2c789e76de04a37db427151f34cbae51be5160109ea3dab6de06b2417d00"
Sep 08 16:38:54.280070 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:54.280025    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ac2c789e76de04a37db427151f34cbae51be5160109ea3dab6de06b2417d00"} err="failed to get container status \"e4ac2c789e76de04a37db427151f34cbae51be5160109ea3dab6de06b2417d00\": rpc error: code = NotFound desc = could not find container \"e4ac2c789e76de04a37db427151f34cbae51be5160109ea3dab6de06b2417d00\": container with ID starting with e4ac2c789e76de04a37db427151f34cbae51be5160109ea3dab6de06b2417d00 not found: ID does not exist"
Sep 08 16:38:54.280070 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:54.280047    2557 scope.go:117] "RemoveContainer" containerID="7da7819e59e09bad726cb15ed5cac4669a785074d5cffcdb452ec9a0d34d72a3"
Sep 08 16:38:54.288880 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:54.288826    2557 scope.go:117] "RemoveContainer" containerID="7da7819e59e09bad726cb15ed5cac4669a785074d5cffcdb452ec9a0d34d72a3"
Sep 08 16:38:54.289138 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:38:54.289119    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da7819e59e09bad726cb15ed5cac4669a785074d5cffcdb452ec9a0d34d72a3\": container with ID starting with 7da7819e59e09bad726cb15ed5cac4669a785074d5cffcdb452ec9a0d34d72a3 not found: ID does not exist" containerID="7da7819e59e09bad726cb15ed5cac4669a785074d5cffcdb452ec9a0d34d72a3"
Sep 08 16:38:54.289194 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:54.289145    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da7819e59e09bad726cb15ed5cac4669a785074d5cffcdb452ec9a0d34d72a3"} err="failed to get container status \"7da7819e59e09bad726cb15ed5cac4669a785074d5cffcdb452ec9a0d34d72a3\": rpc error: code = NotFound desc = could not find container \"7da7819e59e09bad726cb15ed5cac4669a785074d5cffcdb452ec9a0d34d72a3\": container with ID starting with 7da7819e59e09bad726cb15ed5cac4669a785074d5cffcdb452ec9a0d34d72a3 not found: ID does not exist"
Sep 08 16:38:54.301277 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:54.301238    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zs6rn" podStartSLOduration=1.301225426 podStartE2EDuration="1.301225426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:38:54.30055346 +0000 UTC m=+518.463524923" watchObservedRunningTime="2026-09-08 16:38:54.301225426 +0000 UTC m=+518.464196874"
Sep 08 16:38:54.302507 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:38:54.302483    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9dflz\" is forbidden: User \"system:node:ip-10-0-139-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-51.ec2.internal' and this object" podUID="1b7a7b74-be24-4f77-8a70-a869b01aa5c3" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz"
Sep 08 16:38:54.304219 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:38:54.304196    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jphqp\" is forbidden: User \"system:node:ip-10-0-139-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-51.ec2.internal' and this object" podUID="bb486435-63bf-4fba-98b9-2b19050e276b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"
Sep 08 16:38:54.305947 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:38:54.305920    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9dflz\" is forbidden: User \"system:node:ip-10-0-139-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-51.ec2.internal' and this object" podUID="1b7a7b74-be24-4f77-8a70-a869b01aa5c3" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9dflz"
Sep 08 16:38:54.329011 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:54.328965    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kp4f4" podStartSLOduration=1.3289509640000001 podStartE2EDuration="1.328950964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:38:54.327633942 +0000 UTC m=+518.490605400" watchObservedRunningTime="2026-09-08 16:38:54.328950964 +0000 UTC m=+518.491922411"
Sep 08 16:38:54.329981 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:38:54.329952    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jphqp\" is forbidden: User \"system:node:ip-10-0-139-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-51.ec2.internal' and this object" podUID="bb486435-63bf-4fba-98b9-2b19050e276b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jphqp"
Sep 08 16:38:54.345803 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:54.345767    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b7a7b74-be24-4f77-8a70-a869b01aa5c3" path="/var/lib/kubelet/pods/1b7a7b74-be24-4f77-8a70-a869b01aa5c3/volumes"
Sep 08 16:38:54.346189 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:38:54.346171    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb486435-63bf-4fba-98b9-2b19050e276b" path="/var/lib/kubelet/pods/bb486435-63bf-4fba-98b9-2b19050e276b/volumes"
Sep 08 16:39:05.287970 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:05.287938    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kp4f4"
Sep 08 16:39:06.292097 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:06.292067    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zs6rn"
Sep 08 16:39:09.356186 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:09.356099    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zs6rn"]
Sep 08 16:39:09.356601 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:09.356330    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zs6rn" podUID="d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a" containerName="manager" containerID="cri-o://c63bb8c5d813c498b4888b79c497a28426c6f4b4ce8ebda9077a92a13d6ea432" gracePeriod=10
Sep 08 16:39:09.639338 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:09.639305    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zs6rn"
Sep 08 16:39:09.667028 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:09.666984    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ff8f8"]
Sep 08 16:39:09.667478 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:09.667443    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a" containerName="manager"
Sep 08 16:39:09.667478 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:09.667465    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a" containerName="manager"
Sep 08 16:39:09.667621 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:09.667560    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a" containerName="manager"
Sep 08 16:39:09.683496 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:09.683467    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ff8f8"
Sep 08 16:39:09.687975 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:09.687948    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ff8f8"]
Sep 08 16:39:09.756827 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:09.756791    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rc7bc\" (UniqueName: \"kubernetes.io/projected/d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a-kube-api-access-rc7bc\") pod \"d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a\" (UID: \"d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a\") "
Sep 08 16:39:09.757040 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:09.756881    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a-extensions-socket-volume\") pod \"d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a\" (UID: \"d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a\") "
Sep 08 16:39:09.757040 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:09.756996    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50c14b2c-7734-4370-bae2-c0ebbccd57f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ff8f8\" (UID: \"50c14b2c-7734-4370-bae2-c0ebbccd57f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ff8f8"
Sep 08 16:39:09.757155 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:09.757130    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ggfb\" (UniqueName: \"kubernetes.io/projected/50c14b2c-7734-4370-bae2-c0ebbccd57f0-kube-api-access-7ggfb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ff8f8\" (UID: \"50c14b2c-7734-4370-bae2-c0ebbccd57f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ff8f8"
Sep 08 16:39:09.757289 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:09.757271    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a" (UID: "d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:39:09.759028 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:09.759005    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a-kube-api-access-rc7bc" (OuterVolumeSpecName: "kube-api-access-rc7bc") pod "d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a" (UID: "d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a"). InnerVolumeSpecName "kube-api-access-rc7bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:39:09.858222 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:09.858124    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ggfb\" (UniqueName: \"kubernetes.io/projected/50c14b2c-7734-4370-bae2-c0ebbccd57f0-kube-api-access-7ggfb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ff8f8\" (UID: \"50c14b2c-7734-4370-bae2-c0ebbccd57f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ff8f8"
Sep 08 16:39:09.858222 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:09.858186    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50c14b2c-7734-4370-bae2-c0ebbccd57f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ff8f8\" (UID: \"50c14b2c-7734-4370-bae2-c0ebbccd57f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ff8f8"
Sep 08 16:39:09.858425 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:09.858259    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc7bc\" (UniqueName: \"kubernetes.io/projected/d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a-kube-api-access-rc7bc\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:39:09.858425 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:09.858273    2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a-extensions-socket-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:39:09.858605 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:09.858585    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50c14b2c-7734-4370-bae2-c0ebbccd57f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ff8f8\" (UID: \"50c14b2c-7734-4370-bae2-c0ebbccd57f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ff8f8"
Sep 08 16:39:09.866766 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:09.866745    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ggfb\" (UniqueName: \"kubernetes.io/projected/50c14b2c-7734-4370-bae2-c0ebbccd57f0-kube-api-access-7ggfb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ff8f8\" (UID: \"50c14b2c-7734-4370-bae2-c0ebbccd57f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ff8f8"
Sep 08 16:39:09.995570 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:09.995534    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ff8f8"
Sep 08 16:39:10.293543 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:10.293519    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ff8f8"]
Sep 08 16:39:10.296197 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:39:10.296168    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c14b2c_7734_4370_bae2_c0ebbccd57f0.slice/crio-61c4e4bb2c37c6b017ce4079f10cd532d6d5bf55c9796dba5cf1510ee8bed10c WatchSource:0}: Error finding container 61c4e4bb2c37c6b017ce4079f10cd532d6d5bf55c9796dba5cf1510ee8bed10c: Status 404 returned error can't find the container with id 61c4e4bb2c37c6b017ce4079f10cd532d6d5bf55c9796dba5cf1510ee8bed10c
Sep 08 16:39:10.360324 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:10.360287    2557 generic.go:356] "Generic (PLEG): container finished" podID="d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a" containerID="c63bb8c5d813c498b4888b79c497a28426c6f4b4ce8ebda9077a92a13d6ea432" exitCode=0
Sep 08 16:39:10.360628 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:10.360343    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zs6rn"
Sep 08 16:39:10.360628 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:10.360363    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zs6rn" event={"ID":"d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a","Type":"ContainerDied","Data":"c63bb8c5d813c498b4888b79c497a28426c6f4b4ce8ebda9077a92a13d6ea432"}
Sep 08 16:39:10.360628 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:10.360452    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zs6rn" event={"ID":"d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a","Type":"ContainerDied","Data":"093ad35dee9eda4f34bd473200a0cfe7a10a6b607945feda2fb4a87221e00000"}
Sep 08 16:39:10.360628 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:10.360481    2557 scope.go:117] "RemoveContainer" containerID="c63bb8c5d813c498b4888b79c497a28426c6f4b4ce8ebda9077a92a13d6ea432"
Sep 08 16:39:10.361939 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:10.361912    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ff8f8" event={"ID":"50c14b2c-7734-4370-bae2-c0ebbccd57f0","Type":"ContainerStarted","Data":"61c4e4bb2c37c6b017ce4079f10cd532d6d5bf55c9796dba5cf1510ee8bed10c"}
Sep 08 16:39:10.369188 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:10.369173    2557 scope.go:117] "RemoveContainer" containerID="c63bb8c5d813c498b4888b79c497a28426c6f4b4ce8ebda9077a92a13d6ea432"
Sep 08 16:39:10.369450 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:39:10.369430    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c63bb8c5d813c498b4888b79c497a28426c6f4b4ce8ebda9077a92a13d6ea432\": container with ID starting with c63bb8c5d813c498b4888b79c497a28426c6f4b4ce8ebda9077a92a13d6ea432 not found: ID does not exist" containerID="c63bb8c5d813c498b4888b79c497a28426c6f4b4ce8ebda9077a92a13d6ea432"
Sep 08 16:39:10.369492 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:10.369459    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c63bb8c5d813c498b4888b79c497a28426c6f4b4ce8ebda9077a92a13d6ea432"} err="failed to get container status \"c63bb8c5d813c498b4888b79c497a28426c6f4b4ce8ebda9077a92a13d6ea432\": rpc error: code = NotFound desc = could not find container \"c63bb8c5d813c498b4888b79c497a28426c6f4b4ce8ebda9077a92a13d6ea432\": container with ID starting with c63bb8c5d813c498b4888b79c497a28426c6f4b4ce8ebda9077a92a13d6ea432 not found: ID does not exist"
Sep 08 16:39:10.378693 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:10.378670    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zs6rn"]
Sep 08 16:39:10.385420 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:10.385398    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zs6rn"]
Sep 08 16:39:11.369179 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:11.369146    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ff8f8" event={"ID":"50c14b2c-7734-4370-bae2-c0ebbccd57f0","Type":"ContainerStarted","Data":"33736d982c82f0360299ced71998f81eb18b440de5829ac946d35fe8a988227e"}
Sep 08 16:39:11.369576 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:11.369283    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ff8f8"
Sep 08 16:39:11.395197 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:11.395146    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ff8f8" podStartSLOduration=2.395129312 podStartE2EDuration="2.395129312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:39:11.392471006 +0000 UTC m=+535.555442450" watchObservedRunningTime="2026-09-08 16:39:11.395129312 +0000 UTC m=+535.558100759"
Sep 08 16:39:12.345313 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:12.345281    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a" path="/var/lib/kubelet/pods/d2b9c1b6-4767-4b47-ba95-77cdb2ef0c3a/volumes"
Sep 08 16:39:22.376086 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:22.376054    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ff8f8"
Sep 08 16:39:40.637742 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:40.637701    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:39:40.640970 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:40.640949    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"
Sep 08 16:39:40.643501 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:40.643481    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 08 16:39:40.648649 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:40.648630    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:39:40.682902 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:40.682871    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:39:40.696406 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:40.696372    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms6jr\" (UniqueName: \"kubernetes.io/projected/706cd63f-fb56-4f60-9840-9a2a1a4a26bb-kube-api-access-ms6jr\") pod \"limitador-limitador-84994f7ffc-kjxk4\" (UID: \"706cd63f-fb56-4f60-9840-9a2a1a4a26bb\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"
Sep 08 16:39:40.696536 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:40.696432    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/706cd63f-fb56-4f60-9840-9a2a1a4a26bb-config-file\") pod \"limitador-limitador-84994f7ffc-kjxk4\" (UID: \"706cd63f-fb56-4f60-9840-9a2a1a4a26bb\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"
Sep 08 16:39:40.797309 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:40.797259    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/706cd63f-fb56-4f60-9840-9a2a1a4a26bb-config-file\") pod \"limitador-limitador-84994f7ffc-kjxk4\" (UID: \"706cd63f-fb56-4f60-9840-9a2a1a4a26bb\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"
Sep 08 16:39:40.797309 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:40.797321    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms6jr\" (UniqueName: \"kubernetes.io/projected/706cd63f-fb56-4f60-9840-9a2a1a4a26bb-kube-api-access-ms6jr\") pod \"limitador-limitador-84994f7ffc-kjxk4\" (UID: \"706cd63f-fb56-4f60-9840-9a2a1a4a26bb\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"
Sep 08 16:39:40.797965 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:40.797941    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/706cd63f-fb56-4f60-9840-9a2a1a4a26bb-config-file\") pod \"limitador-limitador-84994f7ffc-kjxk4\" (UID: \"706cd63f-fb56-4f60-9840-9a2a1a4a26bb\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"
Sep 08 16:39:40.805797 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:40.805772    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms6jr\" (UniqueName: \"kubernetes.io/projected/706cd63f-fb56-4f60-9840-9a2a1a4a26bb-kube-api-access-ms6jr\") pod \"limitador-limitador-84994f7ffc-kjxk4\" (UID: \"706cd63f-fb56-4f60-9840-9a2a1a4a26bb\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"
Sep 08 16:39:40.951487 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:40.951455    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"
Sep 08 16:39:41.224509 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:41.224484    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:39:41.227018 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:39:41.226993    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod706cd63f_fb56_4f60_9840_9a2a1a4a26bb.slice/crio-db2f7648d8f7c99009703eb15ea06504f00a8ac7193e73a151050b62881db577 WatchSource:0}: Error finding container db2f7648d8f7c99009703eb15ea06504f00a8ac7193e73a151050b62881db577: Status 404 returned error can't find the container with id db2f7648d8f7c99009703eb15ea06504f00a8ac7193e73a151050b62881db577
Sep 08 16:39:41.498269 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:41.498188    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-kjxk4" event={"ID":"706cd63f-fb56-4f60-9840-9a2a1a4a26bb","Type":"ContainerStarted","Data":"db2f7648d8f7c99009703eb15ea06504f00a8ac7193e73a151050b62881db577"}
Sep 08 16:39:42.504197 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:42.504161    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-kjxk4" event={"ID":"706cd63f-fb56-4f60-9840-9a2a1a4a26bb","Type":"ContainerStarted","Data":"e241dbadea1e9ec53146c23510c8f4d6e8a237288ee99395cfded7435f0b4ad6"}
Sep 08 16:39:42.504593 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:42.504298    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"
Sep 08 16:39:42.521712 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:42.521666    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-kjxk4" podStartSLOduration=1.662008347 podStartE2EDuration="2.521649501s" totalImagesPullingTime="859.641154ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:39:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:39:42.520819582 +0000 UTC m=+566.683791042" watchObservedRunningTime="2026-09-08 16:39:42.521649501 +0000 UTC m=+566.684620949"
Sep 08 16:39:53.510090 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:39:53.510060    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"
Sep 08 16:40:58.599185 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:40:58.599154    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7vgjf"]
Sep 08 16:40:58.602212 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:40:58.602194    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-7vgjf"
Sep 08 16:40:58.604478 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:40:58.604453    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 08 16:40:58.604598 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:40:58.604491    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2xqfl\""
Sep 08 16:40:58.617564 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:40:58.617534    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7vgjf"]
Sep 08 16:40:58.676236 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:40:58.676203    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:40:58.705251 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:40:58.705223    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050c1b6a-d113-4083-adb8-b94cb80b1f5c-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7vgjf\" (UID: \"050c1b6a-d113-4083-adb8-b94cb80b1f5c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7vgjf"
Sep 08 16:40:58.705415 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:40:58.705289    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zrq4\" (UniqueName: \"kubernetes.io/projected/050c1b6a-d113-4083-adb8-b94cb80b1f5c-kube-api-access-7zrq4\") pod \"payload-processing-5b54b7fbb7-7vgjf\" (UID: \"050c1b6a-d113-4083-adb8-b94cb80b1f5c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7vgjf"
Sep 08 16:40:58.805805 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:40:58.805767    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zrq4\" (UniqueName: \"kubernetes.io/projected/050c1b6a-d113-4083-adb8-b94cb80b1f5c-kube-api-access-7zrq4\") pod \"payload-processing-5b54b7fbb7-7vgjf\" (UID: \"050c1b6a-d113-4083-adb8-b94cb80b1f5c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7vgjf"
Sep 08 16:40:58.805989 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:40:58.805853    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050c1b6a-d113-4083-adb8-b94cb80b1f5c-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7vgjf\" (UID: \"050c1b6a-d113-4083-adb8-b94cb80b1f5c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7vgjf"
Sep 08 16:40:58.806485 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:40:58.806465    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050c1b6a-d113-4083-adb8-b94cb80b1f5c-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7vgjf\" (UID: \"050c1b6a-d113-4083-adb8-b94cb80b1f5c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7vgjf"
Sep 08 16:40:58.814735 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:40:58.814712    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zrq4\" (UniqueName: \"kubernetes.io/projected/050c1b6a-d113-4083-adb8-b94cb80b1f5c-kube-api-access-7zrq4\") pod \"payload-processing-5b54b7fbb7-7vgjf\" (UID: \"050c1b6a-d113-4083-adb8-b94cb80b1f5c\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7vgjf"
Sep 08 16:40:58.911973 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:40:58.911860    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-7vgjf"
Sep 08 16:40:59.180708 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:40:59.180661    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7vgjf"]
Sep 08 16:40:59.823329 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:40:59.823293    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-7vgjf" event={"ID":"050c1b6a-d113-4083-adb8-b94cb80b1f5c","Type":"ContainerStarted","Data":"3cf6ca647c2334fff7b17fb237bf7b275fc0b4515cf90e0d90f84289491bbe66"}
Sep 08 16:41:01.836195 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:01.836101    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-7vgjf" event={"ID":"050c1b6a-d113-4083-adb8-b94cb80b1f5c","Type":"ContainerStarted","Data":"1cb512d22f6330a365d26bfbd3f714e464629825c0ef1769e70cf2e7b7b6ed3e"}
Sep 08 16:41:01.836603 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:01.836231    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-7vgjf"
Sep 08 16:41:01.853317 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:01.853232    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-7vgjf" podStartSLOduration=1.522309677 podStartE2EDuration="3.853207821s" totalImagesPullingTime="2.330898144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:40:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:41:01.851944034 +0000 UTC m=+646.014915483" watchObservedRunningTime="2026-09-08 16:41:01.853207821 +0000 UTC m=+646.016179273"
Sep 08 16:41:12.843029 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:12.842996    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-7vgjf"
Sep 08 16:41:16.734911 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:16.734876    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:41:31.531005 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:31.530969    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:41:39.627655 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:39.627620    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:41:46.830240 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:46.830202    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:41:50.339083 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.339049    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"]
Sep 08 16:41:50.342447 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.342425    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"
Sep 08 16:41:50.344930 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.344898    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zk588\""
Sep 08 16:41:50.344930 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.344919    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 08 16:41:50.345849 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.345818    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 08 16:41:50.346003 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.345827    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 08 16:41:50.351357 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.351335    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"]
Sep 08 16:41:50.431270 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.431236    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/362b4cb1-2d1e-4fdf-9659-56f0f7cba944-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84\" (UID: \"362b4cb1-2d1e-4fdf-9659-56f0f7cba944\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"
Sep 08 16:41:50.431443 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.431284    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/362b4cb1-2d1e-4fdf-9659-56f0f7cba944-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84\" (UID: \"362b4cb1-2d1e-4fdf-9659-56f0f7cba944\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"
Sep 08 16:41:50.431443 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.431300    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/362b4cb1-2d1e-4fdf-9659-56f0f7cba944-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84\" (UID: \"362b4cb1-2d1e-4fdf-9659-56f0f7cba944\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"
Sep 08 16:41:50.431443 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.431315    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/362b4cb1-2d1e-4fdf-9659-56f0f7cba944-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84\" (UID: \"362b4cb1-2d1e-4fdf-9659-56f0f7cba944\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"
Sep 08 16:41:50.431443 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.431390    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/362b4cb1-2d1e-4fdf-9659-56f0f7cba944-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84\" (UID: \"362b4cb1-2d1e-4fdf-9659-56f0f7cba944\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"
Sep 08 16:41:50.431585 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.431440    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbfrn\" (UniqueName: \"kubernetes.io/projected/362b4cb1-2d1e-4fdf-9659-56f0f7cba944-kube-api-access-sbfrn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84\" (UID: \"362b4cb1-2d1e-4fdf-9659-56f0f7cba944\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"
Sep 08 16:41:50.532117 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.532083    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/362b4cb1-2d1e-4fdf-9659-56f0f7cba944-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84\" (UID: \"362b4cb1-2d1e-4fdf-9659-56f0f7cba944\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"
Sep 08 16:41:50.532286 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.532126    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbfrn\" (UniqueName: \"kubernetes.io/projected/362b4cb1-2d1e-4fdf-9659-56f0f7cba944-kube-api-access-sbfrn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84\" (UID: \"362b4cb1-2d1e-4fdf-9659-56f0f7cba944\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"
Sep 08 16:41:50.532286 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.532255    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/362b4cb1-2d1e-4fdf-9659-56f0f7cba944-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84\" (UID: \"362b4cb1-2d1e-4fdf-9659-56f0f7cba944\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"
Sep 08 16:41:50.532408 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.532336    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/362b4cb1-2d1e-4fdf-9659-56f0f7cba944-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84\" (UID: \"362b4cb1-2d1e-4fdf-9659-56f0f7cba944\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"
Sep 08 16:41:50.532408 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.532365    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/362b4cb1-2d1e-4fdf-9659-56f0f7cba944-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84\" (UID: \"362b4cb1-2d1e-4fdf-9659-56f0f7cba944\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"
Sep 08 16:41:50.532408 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.532390    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/362b4cb1-2d1e-4fdf-9659-56f0f7cba944-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84\" (UID: \"362b4cb1-2d1e-4fdf-9659-56f0f7cba944\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"
Sep 08 16:41:50.532550 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.532454    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/362b4cb1-2d1e-4fdf-9659-56f0f7cba944-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84\" (UID: \"362b4cb1-2d1e-4fdf-9659-56f0f7cba944\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"
Sep 08 16:41:50.532606 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.532578    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/362b4cb1-2d1e-4fdf-9659-56f0f7cba944-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84\" (UID: \"362b4cb1-2d1e-4fdf-9659-56f0f7cba944\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"
Sep 08 16:41:50.532721 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.532699    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/362b4cb1-2d1e-4fdf-9659-56f0f7cba944-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84\" (UID: \"362b4cb1-2d1e-4fdf-9659-56f0f7cba944\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"
Sep 08 16:41:50.534709 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.534682    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/362b4cb1-2d1e-4fdf-9659-56f0f7cba944-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84\" (UID: \"362b4cb1-2d1e-4fdf-9659-56f0f7cba944\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"
Sep 08 16:41:50.534975 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.534960    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/362b4cb1-2d1e-4fdf-9659-56f0f7cba944-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84\" (UID: \"362b4cb1-2d1e-4fdf-9659-56f0f7cba944\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"
Sep 08 16:41:50.540064 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.540044    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbfrn\" (UniqueName: \"kubernetes.io/projected/362b4cb1-2d1e-4fdf-9659-56f0f7cba944-kube-api-access-sbfrn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84\" (UID: \"362b4cb1-2d1e-4fdf-9659-56f0f7cba944\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"
Sep 08 16:41:50.654192 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.654095    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"
Sep 08 16:41:50.929252 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.929078    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"]
Sep 08 16:41:50.931772 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:41:50.931740    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362b4cb1_2d1e_4fdf_9659_56f0f7cba944.slice/crio-4cd836c229d61b70866363fd3b73c04491e8bb69452d682eba7eb4c2c8a6d4e2 WatchSource:0}: Error finding container 4cd836c229d61b70866363fd3b73c04491e8bb69452d682eba7eb4c2c8a6d4e2: Status 404 returned error can't find the container with id 4cd836c229d61b70866363fd3b73c04491e8bb69452d682eba7eb4c2c8a6d4e2
Sep 08 16:41:50.934951 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:50.933431    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 16:41:51.053304 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:51.053262    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84" event={"ID":"362b4cb1-2d1e-4fdf-9659-56f0f7cba944","Type":"ContainerStarted","Data":"4cd836c229d61b70866363fd3b73c04491e8bb69452d682eba7eb4c2c8a6d4e2"}
Sep 08 16:41:51.726491 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:51.726457    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:41:52.059668 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:52.059631    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84" event={"ID":"362b4cb1-2d1e-4fdf-9659-56f0f7cba944","Type":"ContainerStarted","Data":"570c802e42f864d5e81424b58c1ee37f11518d3e601a530d06a0e20a1acab464"}
Sep 08 16:41:52.079699 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:52.079644    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84" podStartSLOduration=1.069702285 podStartE2EDuration="2.079623143s" totalImagesPullingTime="1.009920858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:41:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:41:52.077411586 +0000 UTC m=+696.240383034" watchObservedRunningTime="2026-09-08 16:41:52.079623143 +0000 UTC m=+696.242594594"
Sep 08 16:41:56.947536 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:41:56.947504    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:42:00.654897 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:42:00.654856    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"
Sep 08 16:42:00.655389 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:42:00.654925    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"
Sep 08 16:42:00.667657 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:42:00.667623    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"
Sep 08 16:42:01.113532 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:42:01.113499    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84"
Sep 08 16:42:03.233090 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:42:03.233042    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:43:20.179047 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:20.179014    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7861-139959211822128"]
Sep 08 16:43:20.182064 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:20.182046    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7861-139959211822128"
Sep 08 16:43:20.184720 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:20.184697    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kkgdp\""
Sep 08 16:43:20.188659 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:20.188637    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7861-139959211822128"]
Sep 08 16:43:20.270524 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:20.270492    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lpzk\" (UniqueName: \"kubernetes.io/projected/88f308f0-fff6-4f05-9860-ec4d6f63bb81-kube-api-access-8lpzk\") pod \"test-curl-7861-139959211822128\" (UID: \"88f308f0-fff6-4f05-9860-ec4d6f63bb81\") " pod="opendatahub/test-curl-7861-139959211822128"
Sep 08 16:43:20.371066 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:20.371023    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lpzk\" (UniqueName: \"kubernetes.io/projected/88f308f0-fff6-4f05-9860-ec4d6f63bb81-kube-api-access-8lpzk\") pod \"test-curl-7861-139959211822128\" (UID: \"88f308f0-fff6-4f05-9860-ec4d6f63bb81\") " pod="opendatahub/test-curl-7861-139959211822128"
Sep 08 16:43:20.379763 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:20.379733    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lpzk\" (UniqueName: \"kubernetes.io/projected/88f308f0-fff6-4f05-9860-ec4d6f63bb81-kube-api-access-8lpzk\") pod \"test-curl-7861-139959211822128\" (UID: \"88f308f0-fff6-4f05-9860-ec4d6f63bb81\") " pod="opendatahub/test-curl-7861-139959211822128"
Sep 08 16:43:20.493070 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:20.493001    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7861-139959211822128"
Sep 08 16:43:20.760394 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:20.760370    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7861-139959211822128"]
Sep 08 16:43:20.762491 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:43:20.762459    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88f308f0_fff6_4f05_9860_ec4d6f63bb81.slice/crio-04d9221529195a59fb7a79bfdca01103d0270598304e6248ca8faec0788f3a9c WatchSource:0}: Error finding container 04d9221529195a59fb7a79bfdca01103d0270598304e6248ca8faec0788f3a9c: Status 404 returned error can't find the container with id 04d9221529195a59fb7a79bfdca01103d0270598304e6248ca8faec0788f3a9c
Sep 08 16:43:21.445009 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:21.444972    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7861-139959211822128" event={"ID":"88f308f0-fff6-4f05-9860-ec4d6f63bb81","Type":"ContainerStarted","Data":"04d9221529195a59fb7a79bfdca01103d0270598304e6248ca8faec0788f3a9c"}
Sep 08 16:43:21.840498 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:21.840426    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:43:22.450995 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:22.450958    2557 generic.go:356] "Generic (PLEG): container finished" podID="88f308f0-fff6-4f05-9860-ec4d6f63bb81" containerID="d541ee601fb062971b63137c1373341682dd500505ae9cce65aa713ed18307f1" exitCode=0
Sep 08 16:43:22.451355 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:22.451042    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7861-139959211822128" event={"ID":"88f308f0-fff6-4f05-9860-ec4d6f63bb81","Type":"ContainerDied","Data":"d541ee601fb062971b63137c1373341682dd500505ae9cce65aa713ed18307f1"}
Sep 08 16:43:23.461367 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.461341    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5"]
Sep 08 16:43:23.464667 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.464650    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5"
Sep 08 16:43:23.467354 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.467335    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c9b098\""
Sep 08 16:43:23.467592 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.467576    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c9b098-dockercfg-nbchx\""
Sep 08 16:43:23.476754 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.476727    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5"]
Sep 08 16:43:23.575491 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.575457    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942"]
Sep 08 16:43:23.579113 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.579094    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942"
Sep 08 16:43:23.583969 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.583921    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7861-139959211822128"
Sep 08 16:43:23.589149 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.589127    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942"]
Sep 08 16:43:23.597706 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.597685    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7zhk\" (UniqueName: \"kubernetes.io/projected/db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d-kube-api-access-h7zhk\") pod \"payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5\" (UID: \"db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5"
Sep 08 16:43:23.597794 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.597740    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5\" (UID: \"db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5"
Sep 08 16:43:23.636495 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.636470    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7861-139959211822128_88f308f0-fff6-4f05-9860-ec4d6f63bb81/test-curl-7861-139959211822128/0.log"
Sep 08 16:43:23.660411 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.660384    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7861-139959211822128"]
Sep 08 16:43:23.664866 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.664845    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7861-139959211822128"]
Sep 08 16:43:23.698541 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.698519    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lpzk\" (UniqueName: \"kubernetes.io/projected/88f308f0-fff6-4f05-9860-ec4d6f63bb81-kube-api-access-8lpzk\") pod \"88f308f0-fff6-4f05-9860-ec4d6f63bb81\" (UID: \"88f308f0-fff6-4f05-9860-ec4d6f63bb81\") "
Sep 08 16:43:23.698641 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.698621    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/776168c6-1e80-46df-bf8a-b66a38f678a6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942\" (UID: \"776168c6-1e80-46df-bf8a-b66a38f678a6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942"
Sep 08 16:43:23.698685 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.698656    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7zhk\" (UniqueName: \"kubernetes.io/projected/db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d-kube-api-access-h7zhk\") pod \"payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5\" (UID: \"db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5"
Sep 08 16:43:23.698719 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.698687    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7qv9\" (UniqueName: \"kubernetes.io/projected/776168c6-1e80-46df-bf8a-b66a38f678a6-kube-api-access-f7qv9\") pod \"payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942\" (UID: \"776168c6-1e80-46df-bf8a-b66a38f678a6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942"
Sep 08 16:43:23.698753 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.698724    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5\" (UID: \"db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5"
Sep 08 16:43:23.699423 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.699394    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5\" (UID: \"db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5"
Sep 08 16:43:23.700757 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.700734    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f308f0-fff6-4f05-9860-ec4d6f63bb81-kube-api-access-8lpzk" (OuterVolumeSpecName: "kube-api-access-8lpzk") pod "88f308f0-fff6-4f05-9860-ec4d6f63bb81" (UID: "88f308f0-fff6-4f05-9860-ec4d6f63bb81"). InnerVolumeSpecName "kube-api-access-8lpzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:43:23.707717 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.707699    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7zhk\" (UniqueName: \"kubernetes.io/projected/db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d-kube-api-access-h7zhk\") pod \"payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5\" (UID: \"db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5"
Sep 08 16:43:23.779773 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.779748    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5"
Sep 08 16:43:23.782782 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.782758    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7861-139959211822128"]
Sep 08 16:43:23.783159 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.783146    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88f308f0-fff6-4f05-9860-ec4d6f63bb81" containerName="test-curl-7861-139959211822128"
Sep 08 16:43:23.783195 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.783162    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="88f308f0-fff6-4f05-9860-ec4d6f63bb81" containerName="test-curl-7861-139959211822128"
Sep 08 16:43:23.783228 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.783213    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="88f308f0-fff6-4f05-9860-ec4d6f63bb81" containerName="test-curl-7861-139959211822128"
Sep 08 16:43:23.786012 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.785997    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7861-139959211822128"
Sep 08 16:43:23.799181 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.799038    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7861-139959211822128"]
Sep 08 16:43:23.799414 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.799389    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/776168c6-1e80-46df-bf8a-b66a38f678a6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942\" (UID: \"776168c6-1e80-46df-bf8a-b66a38f678a6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942"
Sep 08 16:43:23.799544 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.799438    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7qv9\" (UniqueName: \"kubernetes.io/projected/776168c6-1e80-46df-bf8a-b66a38f678a6-kube-api-access-f7qv9\") pod \"payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942\" (UID: \"776168c6-1e80-46df-bf8a-b66a38f678a6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942"
Sep 08 16:43:23.799544 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.799522    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lpzk\" (UniqueName: \"kubernetes.io/projected/88f308f0-fff6-4f05-9860-ec4d6f63bb81-kube-api-access-8lpzk\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:43:23.800166 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.800125    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/776168c6-1e80-46df-bf8a-b66a38f678a6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942\" (UID: \"776168c6-1e80-46df-bf8a-b66a38f678a6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942"
Sep 08 16:43:23.808599 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.808577    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7qv9\" (UniqueName: \"kubernetes.io/projected/776168c6-1e80-46df-bf8a-b66a38f678a6-kube-api-access-f7qv9\") pod \"payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942\" (UID: \"776168c6-1e80-46df-bf8a-b66a38f678a6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942"
Sep 08 16:43:23.891193 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.891161    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942"
Sep 08 16:43:23.900642 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:23.900565    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltr28\" (UniqueName: \"kubernetes.io/projected/11d744b8-1e90-4b50-913e-1eb7661db664-kube-api-access-ltr28\") pod \"test-curl-7861-139959211822128\" (UID: \"11d744b8-1e90-4b50-913e-1eb7661db664\") " pod="opendatahub/test-curl-7861-139959211822128"
Sep 08 16:43:24.001454 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:24.001381    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltr28\" (UniqueName: \"kubernetes.io/projected/11d744b8-1e90-4b50-913e-1eb7661db664-kube-api-access-ltr28\") pod \"test-curl-7861-139959211822128\" (UID: \"11d744b8-1e90-4b50-913e-1eb7661db664\") " pod="opendatahub/test-curl-7861-139959211822128"
Sep 08 16:43:24.009077 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:24.009052    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltr28\" (UniqueName: \"kubernetes.io/projected/11d744b8-1e90-4b50-913e-1eb7661db664-kube-api-access-ltr28\") pod \"test-curl-7861-139959211822128\" (UID: \"11d744b8-1e90-4b50-913e-1eb7661db664\") " pod="opendatahub/test-curl-7861-139959211822128"
Sep 08 16:43:24.053234 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:24.053192    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5"]
Sep 08 16:43:24.053578 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:43:24.053553    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb2ce4c5_0c6f_4c5e_9929_6bc44d3c980d.slice/crio-f65328ad34607857498ec2314b047778bc52eb36839be4e7aa2fdf5b974d4958 WatchSource:0}: Error finding container f65328ad34607857498ec2314b047778bc52eb36839be4e7aa2fdf5b974d4958: Status 404 returned error can't find the container with id f65328ad34607857498ec2314b047778bc52eb36839be4e7aa2fdf5b974d4958
Sep 08 16:43:24.104219 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:24.104106    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7861-139959211822128"
Sep 08 16:43:24.163588 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:24.162798    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942"]
Sep 08 16:43:24.165470 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:43:24.165421    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776168c6_1e80_46df_bf8a_b66a38f678a6.slice/crio-6a996591e679a7c2f09b92b266fbacb7deb2db9038d4163dc866002bf39079ef WatchSource:0}: Error finding container 6a996591e679a7c2f09b92b266fbacb7deb2db9038d4163dc866002bf39079ef: Status 404 returned error can't find the container with id 6a996591e679a7c2f09b92b266fbacb7deb2db9038d4163dc866002bf39079ef
Sep 08 16:43:24.346637 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:24.346559    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88f308f0-fff6-4f05-9860-ec4d6f63bb81" path="/var/lib/kubelet/pods/88f308f0-fff6-4f05-9860-ec4d6f63bb81/volumes"
Sep 08 16:43:24.374745 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:24.374684    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7861-139959211822128"]
Sep 08 16:43:24.377165 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:43:24.377139    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11d744b8_1e90_4b50_913e_1eb7661db664.slice/crio-2adfe8b881c61d3a1c05461db2cf324ee202dca62708898c9eba4928e4151845 WatchSource:0}: Error finding container 2adfe8b881c61d3a1c05461db2cf324ee202dca62708898c9eba4928e4151845: Status 404 returned error can't find the container with id 2adfe8b881c61d3a1c05461db2cf324ee202dca62708898c9eba4928e4151845
Sep 08 16:43:24.461333 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:24.461294    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7861-139959211822128" event={"ID":"11d744b8-1e90-4b50-913e-1eb7661db664","Type":"ContainerStarted","Data":"2adfe8b881c61d3a1c05461db2cf324ee202dca62708898c9eba4928e4151845"}
Sep 08 16:43:24.462651 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:24.462635    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7861-139959211822128"
Sep 08 16:43:24.462651 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:24.462644    2557 scope.go:117] "RemoveContainer" containerID="d541ee601fb062971b63137c1373341682dd500505ae9cce65aa713ed18307f1"
Sep 08 16:43:24.464260 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:24.464233    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942" event={"ID":"776168c6-1e80-46df-bf8a-b66a38f678a6","Type":"ContainerStarted","Data":"37dae651d7c0fddf48cb8d142a361ddf510f796626f5a4356c3fab17fffaf25e"}
Sep 08 16:43:24.464363 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:24.464291    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942" event={"ID":"776168c6-1e80-46df-bf8a-b66a38f678a6","Type":"ContainerStarted","Data":"6a996591e679a7c2f09b92b266fbacb7deb2db9038d4163dc866002bf39079ef"}
Sep 08 16:43:24.464363 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:24.464326    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942"
Sep 08 16:43:24.466216 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:24.466121    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5" event={"ID":"db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d","Type":"ContainerStarted","Data":"4c637d572cb5236ff12f8900c94ed1bd07f8eec425becf403df2de76177683e0"}
Sep 08 16:43:24.466216 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:24.466170    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5" event={"ID":"db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d","Type":"ContainerStarted","Data":"f65328ad34607857498ec2314b047778bc52eb36839be4e7aa2fdf5b974d4958"}
Sep 08 16:43:24.466376 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:24.466222    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5"
Sep 08 16:43:24.483710 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:24.483662    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942" podStartSLOduration=1.483643264 podStartE2EDuration="1.483643264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:43:24.482934422 +0000 UTC m=+788.645905871" watchObservedRunningTime="2026-09-08 16:43:24.483643264 +0000 UTC m=+788.646614712"
Sep 08 16:43:24.505266 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:24.505196    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5" podStartSLOduration=1.505177775 podStartE2EDuration="1.505177775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:43:24.504627589 +0000 UTC m=+788.667599038" watchObservedRunningTime="2026-09-08 16:43:24.505177775 +0000 UTC m=+788.668149253"
Sep 08 16:43:25.473103 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:25.473067    2557 generic.go:356] "Generic (PLEG): container finished" podID="11d744b8-1e90-4b50-913e-1eb7661db664" containerID="d5f7587033f949373cd61b8510343b78b615919d377d9535116485d624da2c1d" exitCode=0
Sep 08 16:43:25.473568 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:25.473157    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7861-139959211822128" event={"ID":"11d744b8-1e90-4b50-913e-1eb7661db664","Type":"ContainerDied","Data":"d5f7587033f949373cd61b8510343b78b615919d377d9535116485d624da2c1d"}
Sep 08 16:43:26.612987 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:26.612962    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7861-139959211822128"
Sep 08 16:43:26.634944 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:26.634916    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7861-139959211822128_11d744b8-1e90-4b50-913e-1eb7661db664/test-curl-7861-139959211822128/0.log"
Sep 08 16:43:26.655760 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:26.655723    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7861-139959211822128"]
Sep 08 16:43:26.657712 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:26.657690    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7861-139959211822128"]
Sep 08 16:43:26.726661 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:26.726632    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltr28\" (UniqueName: \"kubernetes.io/projected/11d744b8-1e90-4b50-913e-1eb7661db664-kube-api-access-ltr28\") pod \"11d744b8-1e90-4b50-913e-1eb7661db664\" (UID: \"11d744b8-1e90-4b50-913e-1eb7661db664\") "
Sep 08 16:43:26.728858 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:26.728819    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d744b8-1e90-4b50-913e-1eb7661db664-kube-api-access-ltr28" (OuterVolumeSpecName: "kube-api-access-ltr28") pod "11d744b8-1e90-4b50-913e-1eb7661db664" (UID: "11d744b8-1e90-4b50-913e-1eb7661db664"). InnerVolumeSpecName "kube-api-access-ltr28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:43:26.828166 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:26.828127    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltr28\" (UniqueName: \"kubernetes.io/projected/11d744b8-1e90-4b50-913e-1eb7661db664-kube-api-access-ltr28\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:43:27.486305 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:27.486277    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7861-139959211822128"
Sep 08 16:43:27.486482 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:27.486290    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2adfe8b881c61d3a1c05461db2cf324ee202dca62708898c9eba4928e4151845"
Sep 08 16:43:27.496825 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:43:27.496796    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7861-139959211822128\" is forbidden: User \"system:node:ip-10-0-139-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-51.ec2.internal' and this object" podUID="11d744b8-1e90-4b50-913e-1eb7661db664" pod="opendatahub/test-curl-7861-139959211822128"
Sep 08 16:43:28.345440 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:28.345401    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11d744b8-1e90-4b50-913e-1eb7661db664" path="/var/lib/kubelet/pods/11d744b8-1e90-4b50-913e-1eb7661db664/volumes"
Sep 08 16:43:31.996311 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:31.996276    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v"]
Sep 08 16:43:31.996883 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:31.996773    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d744b8-1e90-4b50-913e-1eb7661db664" containerName="test-curl-7861-139959211822128"
Sep 08 16:43:31.996883 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:31.996793    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d744b8-1e90-4b50-913e-1eb7661db664" containerName="test-curl-7861-139959211822128"
Sep 08 16:43:31.997008 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:31.996921    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="11d744b8-1e90-4b50-913e-1eb7661db664" containerName="test-curl-7861-139959211822128"
Sep 08 16:43:32.000077 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:32.000058    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v"
Sep 08 16:43:32.003653 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:32.003632    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4c81c932-dockercfg-rl2x7\""
Sep 08 16:43:32.003746 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:32.003656    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4c81c932\""
Sep 08 16:43:32.011138 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:32.011113    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v"]
Sep 08 16:43:32.090192 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:32.090153    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8"]
Sep 08 16:43:32.093376 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:32.093358    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8"
Sep 08 16:43:32.113124 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:32.113097    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8"]
Sep 08 16:43:32.175171 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:32.175138    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86683324-78e7-4949-89c6-92162a083ff3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v\" (UID: \"86683324-78e7-4949-89c6-92162a083ff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v"
Sep 08 16:43:32.175321 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:32.175183    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcdtr\" (UniqueName: \"kubernetes.io/projected/86683324-78e7-4949-89c6-92162a083ff3-kube-api-access-tcdtr\") pod \"payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v\" (UID: \"86683324-78e7-4949-89c6-92162a083ff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v"
Sep 08 16:43:32.275926 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:32.275852    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcdtr\" (UniqueName: \"kubernetes.io/projected/86683324-78e7-4949-89c6-92162a083ff3-kube-api-access-tcdtr\") pod \"payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v\" (UID: \"86683324-78e7-4949-89c6-92162a083ff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v"
Sep 08 16:43:32.275926 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:32.275918    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq7pn\" (UniqueName: \"kubernetes.io/projected/5beeb590-3aec-4c9f-9614-fd66209585f1-kube-api-access-dq7pn\") pod \"payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8\" (UID: \"5beeb590-3aec-4c9f-9614-fd66209585f1\") " pod="openshift-ingress/payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8"
Sep 08 16:43:32.276098 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:32.275953    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5beeb590-3aec-4c9f-9614-fd66209585f1-plugins-config-volume\") pod \"payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8\" (UID: \"5beeb590-3aec-4c9f-9614-fd66209585f1\") " pod="openshift-ingress/payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8"
Sep 08 16:43:32.276098 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:32.276014    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86683324-78e7-4949-89c6-92162a083ff3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v\" (UID: \"86683324-78e7-4949-89c6-92162a083ff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v"
Sep 08 16:43:32.276733 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:32.276713    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86683324-78e7-4949-89c6-92162a083ff3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v\" (UID: \"86683324-78e7-4949-89c6-92162a083ff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v"
Sep 08 16:43:32.285023 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:32.284992    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcdtr\" (UniqueName: \"kubernetes.io/projected/86683324-78e7-4949-89c6-92162a083ff3-kube-api-access-tcdtr\") pod \"payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v\" (UID: \"86683324-78e7-4949-89c6-92162a083ff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v"
Sep 08 16:43:32.309902 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:32.309869    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v"
Sep 08 16:43:32.377134 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:32.377107    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq7pn\" (UniqueName: \"kubernetes.io/projected/5beeb590-3aec-4c9f-9614-fd66209585f1-kube-api-access-dq7pn\") pod \"payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8\" (UID: \"5beeb590-3aec-4c9f-9614-fd66209585f1\") " pod="openshift-ingress/payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8"
Sep 08 16:43:32.377278 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:32.377141    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5beeb590-3aec-4c9f-9614-fd66209585f1-plugins-config-volume\") pod \"payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8\" (UID: \"5beeb590-3aec-4c9f-9614-fd66209585f1\") " pod="openshift-ingress/payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8"
Sep 08 16:43:32.377939 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:32.377918    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5beeb590-3aec-4c9f-9614-fd66209585f1-plugins-config-volume\") pod \"payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8\" (UID: \"5beeb590-3aec-4c9f-9614-fd66209585f1\") " pod="openshift-ingress/payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8"
Sep 08 16:43:32.386476 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:32.386438    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq7pn\" (UniqueName: \"kubernetes.io/projected/5beeb590-3aec-4c9f-9614-fd66209585f1-kube-api-access-dq7pn\") pod \"payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8\" (UID: \"5beeb590-3aec-4c9f-9614-fd66209585f1\") " pod="openshift-ingress/payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8"
Sep 08 16:43:32.403539 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:32.403513    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8"
Sep 08 16:43:32.584356 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:32.584326    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v"]
Sep 08 16:43:32.586084 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:43:32.586060    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86683324_78e7_4949_89c6_92162a083ff3.slice/crio-bfb569da23f3b41f3ccc21f3f33fab99b1fb238e47af3618371d2ac2ca337bb6 WatchSource:0}: Error finding container bfb569da23f3b41f3ccc21f3f33fab99b1fb238e47af3618371d2ac2ca337bb6: Status 404 returned error can't find the container with id bfb569da23f3b41f3ccc21f3f33fab99b1fb238e47af3618371d2ac2ca337bb6
Sep 08 16:43:32.680441 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:32.680379    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8"]
Sep 08 16:43:32.682864 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:43:32.682815    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5beeb590_3aec_4c9f_9614_fd66209585f1.slice/crio-316e54ecb97a851282fb75a9e23e348e396cf85ac7154502becdfebad7fd7be6 WatchSource:0}: Error finding container 316e54ecb97a851282fb75a9e23e348e396cf85ac7154502becdfebad7fd7be6: Status 404 returned error can't find the container with id 316e54ecb97a851282fb75a9e23e348e396cf85ac7154502becdfebad7fd7be6
Sep 08 16:43:33.307333 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.307303    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx"]
Sep 08 16:43:33.311072 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.311050    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx"
Sep 08 16:43:33.314453 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.314432    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-4cea93\""
Sep 08 16:43:33.322905 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.322884    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx"]
Sep 08 16:43:33.368742 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.368711    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq"]
Sep 08 16:43:33.372429 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.372411    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq"
Sep 08 16:43:33.375030 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.375012    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-4cea93-dockercfg-vbbgd\""
Sep 08 16:43:33.379976 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.379958    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq"]
Sep 08 16:43:33.487988 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.487960    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25e24a89-d3b7-4413-95a0-035f73c78cb5-plugins-config-volume\") pod \"payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx\" (UID: \"25e24a89-d3b7-4413-95a0-035f73c78cb5\") " pod="openshift-ingress/payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx"
Sep 08 16:43:33.487988 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.488001    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgwmh\" (UniqueName: \"kubernetes.io/projected/25e24a89-d3b7-4413-95a0-035f73c78cb5-kube-api-access-mgwmh\") pod \"payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx\" (UID: \"25e24a89-d3b7-4413-95a0-035f73c78cb5\") " pod="openshift-ingress/payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx"
Sep 08 16:43:33.488198 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.488073    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvjzb\" (UniqueName: \"kubernetes.io/projected/67b7f626-d8af-4970-976b-f30ff734dba1-kube-api-access-mvjzb\") pod \"payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq\" (UID: \"67b7f626-d8af-4970-976b-f30ff734dba1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq"
Sep 08 16:43:33.488198 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.488143    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b7f626-d8af-4970-976b-f30ff734dba1-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq\" (UID: \"67b7f626-d8af-4970-976b-f30ff734dba1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq"
Sep 08 16:43:33.525156 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.525118    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v" event={"ID":"86683324-78e7-4949-89c6-92162a083ff3","Type":"ContainerStarted","Data":"c117c5fafda3135c9daa61dcda1a4d53c5cb060469213bd238bfd20068faccf0"}
Sep 08 16:43:33.525343 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.525180    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v" event={"ID":"86683324-78e7-4949-89c6-92162a083ff3","Type":"ContainerStarted","Data":"bfb569da23f3b41f3ccc21f3f33fab99b1fb238e47af3618371d2ac2ca337bb6"}
Sep 08 16:43:33.525343 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.525214    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v"
Sep 08 16:43:33.526581 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.526556    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8" event={"ID":"5beeb590-3aec-4c9f-9614-fd66209585f1","Type":"ContainerStarted","Data":"e96d9707907e25ae2a1705e98cec59929d2c89d2bc17469b95667e013b6ffa02"}
Sep 08 16:43:33.526728 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.526602    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8" event={"ID":"5beeb590-3aec-4c9f-9614-fd66209585f1","Type":"ContainerStarted","Data":"316e54ecb97a851282fb75a9e23e348e396cf85ac7154502becdfebad7fd7be6"}
Sep 08 16:43:33.526728 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.526638    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8"
Sep 08 16:43:33.544039 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.543993    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v" podStartSLOduration=2.5439780819999998 podStartE2EDuration="2.543978082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:43:33.541794129 +0000 UTC m=+797.704765578" watchObservedRunningTime="2026-09-08 16:43:33.543978082 +0000 UTC m=+797.706949529"
Sep 08 16:43:33.559449 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.559371    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8" podStartSLOduration=1.559357783 podStartE2EDuration="1.559357783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:43:33.558398527 +0000 UTC m=+797.721369976" watchObservedRunningTime="2026-09-08 16:43:33.559357783 +0000 UTC m=+797.722329230"
Sep 08 16:43:33.588872 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.588818    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvjzb\" (UniqueName: \"kubernetes.io/projected/67b7f626-d8af-4970-976b-f30ff734dba1-kube-api-access-mvjzb\") pod \"payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq\" (UID: \"67b7f626-d8af-4970-976b-f30ff734dba1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq"
Sep 08 16:43:33.589012 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.588900    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b7f626-d8af-4970-976b-f30ff734dba1-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq\" (UID: \"67b7f626-d8af-4970-976b-f30ff734dba1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq"
Sep 08 16:43:33.589012 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.588950    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25e24a89-d3b7-4413-95a0-035f73c78cb5-plugins-config-volume\") pod \"payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx\" (UID: \"25e24a89-d3b7-4413-95a0-035f73c78cb5\") " pod="openshift-ingress/payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx"
Sep 08 16:43:33.589012 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.588990    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgwmh\" (UniqueName: \"kubernetes.io/projected/25e24a89-d3b7-4413-95a0-035f73c78cb5-kube-api-access-mgwmh\") pod \"payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx\" (UID: \"25e24a89-d3b7-4413-95a0-035f73c78cb5\") " pod="openshift-ingress/payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx"
Sep 08 16:43:33.589626 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.589601    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b7f626-d8af-4970-976b-f30ff734dba1-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq\" (UID: \"67b7f626-d8af-4970-976b-f30ff734dba1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq"
Sep 08 16:43:33.589813 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.589792    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25e24a89-d3b7-4413-95a0-035f73c78cb5-plugins-config-volume\") pod \"payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx\" (UID: \"25e24a89-d3b7-4413-95a0-035f73c78cb5\") " pod="openshift-ingress/payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx"
Sep 08 16:43:33.596940 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.596913    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgwmh\" (UniqueName: \"kubernetes.io/projected/25e24a89-d3b7-4413-95a0-035f73c78cb5-kube-api-access-mgwmh\") pod \"payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx\" (UID: \"25e24a89-d3b7-4413-95a0-035f73c78cb5\") " pod="openshift-ingress/payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx"
Sep 08 16:43:33.597120 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.597099    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvjzb\" (UniqueName: \"kubernetes.io/projected/67b7f626-d8af-4970-976b-f30ff734dba1-kube-api-access-mvjzb\") pod \"payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq\" (UID: \"67b7f626-d8af-4970-976b-f30ff734dba1\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq"
Sep 08 16:43:33.622801 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.622779    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx"
Sep 08 16:43:33.686375 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.686249    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq"
Sep 08 16:43:33.901057 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.901032    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx"]
Sep 08 16:43:33.903097 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:43:33.903074    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e24a89_d3b7_4413_95a0_035f73c78cb5.slice/crio-5ced992e5439d2e25dd7fb63b491ab38bbd913c18d6aeeff1cc17bc80ca78367 WatchSource:0}: Error finding container 5ced992e5439d2e25dd7fb63b491ab38bbd913c18d6aeeff1cc17bc80ca78367: Status 404 returned error can't find the container with id 5ced992e5439d2e25dd7fb63b491ab38bbd913c18d6aeeff1cc17bc80ca78367
Sep 08 16:43:33.961163 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:33.961139    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq"]
Sep 08 16:43:33.962922 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:43:33.962898    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b7f626_d8af_4970_976b_f30ff734dba1.slice/crio-e9f9c6c43ca9a90d7d9c8e5c983a3981d09ddd9408b6328ed5c9bf12f14ec2ce WatchSource:0}: Error finding container e9f9c6c43ca9a90d7d9c8e5c983a3981d09ddd9408b6328ed5c9bf12f14ec2ce: Status 404 returned error can't find the container with id e9f9c6c43ca9a90d7d9c8e5c983a3981d09ddd9408b6328ed5c9bf12f14ec2ce
Sep 08 16:43:34.532201 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:34.532169    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq" event={"ID":"67b7f626-d8af-4970-976b-f30ff734dba1","Type":"ContainerStarted","Data":"d572f0ddfdfaa70bee05bd86361659652b86da5d79db7fcc2c9eafb66779eeb6"}
Sep 08 16:43:34.532614 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:34.532240    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq"
Sep 08 16:43:34.532614 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:34.532268    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq" event={"ID":"67b7f626-d8af-4970-976b-f30ff734dba1","Type":"ContainerStarted","Data":"e9f9c6c43ca9a90d7d9c8e5c983a3981d09ddd9408b6328ed5c9bf12f14ec2ce"}
Sep 08 16:43:34.533785 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:34.533750    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx" event={"ID":"25e24a89-d3b7-4413-95a0-035f73c78cb5","Type":"ContainerStarted","Data":"2d706540e10b6d8576a822f269040a15931c9cb1775a7b760adb96184d3ca9ba"}
Sep 08 16:43:34.533919 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:34.533801    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx" event={"ID":"25e24a89-d3b7-4413-95a0-035f73c78cb5","Type":"ContainerStarted","Data":"5ced992e5439d2e25dd7fb63b491ab38bbd913c18d6aeeff1cc17bc80ca78367"}
Sep 08 16:43:34.549787 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:34.549743    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq" podStartSLOduration=1.549726859 podStartE2EDuration="1.549726859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:43:34.549090506 +0000 UTC m=+798.712061953" watchObservedRunningTime="2026-09-08 16:43:34.549726859 +0000 UTC m=+798.712698330"
Sep 08 16:43:34.566760 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:34.566716    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx" podStartSLOduration=1.56670425 podStartE2EDuration="1.56670425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:43:34.564759003 +0000 UTC m=+798.727730452" watchObservedRunningTime="2026-09-08 16:43:34.56670425 +0000 UTC m=+798.729675741"
Sep 08 16:43:35.476903 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:35.476873    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5"
Sep 08 16:43:35.477088 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:35.476942    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942"
Sep 08 16:43:35.539102 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:35.539071    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx"
Sep 08 16:43:37.231526 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:37.231448    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:43:37.877938 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:37.877899    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"]
Sep 08 16:43:37.881538 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:37.881515    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:43:37.885315 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:37.885297    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-1be25697-kserve-self-signed-certs\""
Sep 08 16:43:37.899903 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:37.899883    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"]
Sep 08 16:43:37.927271 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:37.927236    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cce2c68-26b9-49b5-920e-021bc699ee5a-home\") pod \"e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf\" (UID: \"4cce2c68-26b9-49b5-920e-021bc699ee5a\") " pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:43:37.927411 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:37.927274    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cce2c68-26b9-49b5-920e-021bc699ee5a-tmp-dir\") pod \"e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf\" (UID: \"4cce2c68-26b9-49b5-920e-021bc699ee5a\") " pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:43:37.927464 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:37.927412    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cce2c68-26b9-49b5-920e-021bc699ee5a-tls-certs\") pod \"e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf\" (UID: \"4cce2c68-26b9-49b5-920e-021bc699ee5a\") " pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:43:37.927464 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:37.927446    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cce2c68-26b9-49b5-920e-021bc699ee5a-dshm\") pod \"e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf\" (UID: \"4cce2c68-26b9-49b5-920e-021bc699ee5a\") " pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:43:37.927548 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:37.927464    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cce2c68-26b9-49b5-920e-021bc699ee5a-model-cache\") pod \"e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf\" (UID: \"4cce2c68-26b9-49b5-920e-021bc699ee5a\") " pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:43:37.927548 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:37.927479    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp2sh\" (UniqueName: \"kubernetes.io/projected/4cce2c68-26b9-49b5-920e-021bc699ee5a-kube-api-access-wp2sh\") pod \"e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf\" (UID: \"4cce2c68-26b9-49b5-920e-021bc699ee5a\") " pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:43:38.028545 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:38.028508    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cce2c68-26b9-49b5-920e-021bc699ee5a-home\") pod \"e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf\" (UID: \"4cce2c68-26b9-49b5-920e-021bc699ee5a\") " pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:43:38.028545 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:38.028543    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cce2c68-26b9-49b5-920e-021bc699ee5a-tmp-dir\") pod \"e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf\" (UID: \"4cce2c68-26b9-49b5-920e-021bc699ee5a\") " pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:43:38.028778 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:38.028613    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cce2c68-26b9-49b5-920e-021bc699ee5a-tls-certs\") pod \"e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf\" (UID: \"4cce2c68-26b9-49b5-920e-021bc699ee5a\") " pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:43:38.028778 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:38.028637    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cce2c68-26b9-49b5-920e-021bc699ee5a-dshm\") pod \"e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf\" (UID: \"4cce2c68-26b9-49b5-920e-021bc699ee5a\") " pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:43:38.028778 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:38.028665    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cce2c68-26b9-49b5-920e-021bc699ee5a-model-cache\") pod \"e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf\" (UID: \"4cce2c68-26b9-49b5-920e-021bc699ee5a\") " pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:43:38.028778 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:38.028690    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp2sh\" (UniqueName: \"kubernetes.io/projected/4cce2c68-26b9-49b5-920e-021bc699ee5a-kube-api-access-wp2sh\") pod \"e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf\" (UID: \"4cce2c68-26b9-49b5-920e-021bc699ee5a\") " pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:43:38.029015 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:38.028967    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cce2c68-26b9-49b5-920e-021bc699ee5a-home\") pod \"e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf\" (UID: \"4cce2c68-26b9-49b5-920e-021bc699ee5a\") " pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:43:38.029070 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:38.029031    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cce2c68-26b9-49b5-920e-021bc699ee5a-tmp-dir\") pod \"e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf\" (UID: \"4cce2c68-26b9-49b5-920e-021bc699ee5a\") " pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:43:38.029139 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:38.029114    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cce2c68-26b9-49b5-920e-021bc699ee5a-model-cache\") pod \"e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf\" (UID: \"4cce2c68-26b9-49b5-920e-021bc699ee5a\") " pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:43:38.031038 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:38.031018    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cce2c68-26b9-49b5-920e-021bc699ee5a-dshm\") pod \"e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf\" (UID: \"4cce2c68-26b9-49b5-920e-021bc699ee5a\") " pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:43:38.031210 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:38.031194    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cce2c68-26b9-49b5-920e-021bc699ee5a-tls-certs\") pod \"e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf\" (UID: \"4cce2c68-26b9-49b5-920e-021bc699ee5a\") " pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:43:38.037721 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:38.037701    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp2sh\" (UniqueName: \"kubernetes.io/projected/4cce2c68-26b9-49b5-920e-021bc699ee5a-kube-api-access-wp2sh\") pod \"e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf\" (UID: \"4cce2c68-26b9-49b5-920e-021bc699ee5a\") " pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:43:38.190789 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:38.190760    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:43:38.464715 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:38.464550    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"]
Sep 08 16:43:38.467435 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:43:38.467409    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cce2c68_26b9_49b5_920e_021bc699ee5a.slice/crio-fd63f2b7e9e534b00990006ff1b08771f5ba3e45128b1561d2d5061b0a1b9ee3 WatchSource:0}: Error finding container fd63f2b7e9e534b00990006ff1b08771f5ba3e45128b1561d2d5061b0a1b9ee3: Status 404 returned error can't find the container with id fd63f2b7e9e534b00990006ff1b08771f5ba3e45128b1561d2d5061b0a1b9ee3
Sep 08 16:43:38.559130 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:38.559097    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf" event={"ID":"4cce2c68-26b9-49b5-920e-021bc699ee5a","Type":"ContainerStarted","Data":"f597f8e522081d457e118cb89fc8ea083cce18c6bb72b01a3db4395ec3459c18"}
Sep 08 16:43:38.559130 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:38.559138    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf" event={"ID":"4cce2c68-26b9-49b5-920e-021bc699ee5a","Type":"ContainerStarted","Data":"fd63f2b7e9e534b00990006ff1b08771f5ba3e45128b1561d2d5061b0a1b9ee3"}
Sep 08 16:43:38.582334 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:38.582281    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf" podStartSLOduration=1.582265528 podStartE2EDuration="1.582265528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:43:38.580535772 +0000 UTC m=+802.743507232" watchObservedRunningTime="2026-09-08 16:43:38.582265528 +0000 UTC m=+802.745236975"
Sep 08 16:43:44.535812 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:44.535780    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v"
Sep 08 16:43:44.536206 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:44.535961    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8"
Sep 08 16:43:45.541605 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:45.541576    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq"
Sep 08 16:43:46.545998 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:46.545974    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx"
Sep 08 16:43:48.191939 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:48.191904    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:43:48.192399 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:48.191952    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:43:48.204534 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:48.204506    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:43:48.625531 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:48.625452    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:43:51.934795 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:51.934756    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:43:52.438351 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:52.438317    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd"]
Sep 08 16:43:52.441780 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:52.441765    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd"
Sep 08 16:43:52.444666 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:52.444644    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-43a905-dockercfg-sjms2\""
Sep 08 16:43:52.444666 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:52.444659    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-43a905\""
Sep 08 16:43:52.454712 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:52.454691    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd"]
Sep 08 16:43:52.504295 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:52.504261    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt"]
Sep 08 16:43:52.507824 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:52.507804    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt"
Sep 08 16:43:52.518104 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:52.518081    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt"]
Sep 08 16:43:52.561707 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:52.561671    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48frh\" (UniqueName: \"kubernetes.io/projected/789ae719-5721-4edd-b05f-b5a312b3ec4c-kube-api-access-48frh\") pod \"payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd\" (UID: \"789ae719-5721-4edd-b05f-b5a312b3ec4c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd"
Sep 08 16:43:52.561904 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:52.561713    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f3c2c9-b701-4d0e-b5ce-047682111980-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt\" (UID: \"b0f3c2c9-b701-4d0e-b5ce-047682111980\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt"
Sep 08 16:43:52.561904 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:52.561797    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ql88\" (UniqueName: \"kubernetes.io/projected/b0f3c2c9-b701-4d0e-b5ce-047682111980-kube-api-access-4ql88\") pod \"payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt\" (UID: \"b0f3c2c9-b701-4d0e-b5ce-047682111980\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt"
Sep 08 16:43:52.561904 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:52.561869    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789ae719-5721-4edd-b05f-b5a312b3ec4c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd\" (UID: \"789ae719-5721-4edd-b05f-b5a312b3ec4c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd"
Sep 08 16:43:52.662791 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:52.662756    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ql88\" (UniqueName: \"kubernetes.io/projected/b0f3c2c9-b701-4d0e-b5ce-047682111980-kube-api-access-4ql88\") pod \"payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt\" (UID: \"b0f3c2c9-b701-4d0e-b5ce-047682111980\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt"
Sep 08 16:43:52.662975 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:52.662800    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789ae719-5721-4edd-b05f-b5a312b3ec4c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd\" (UID: \"789ae719-5721-4edd-b05f-b5a312b3ec4c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd"
Sep 08 16:43:52.662975 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:52.662892    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48frh\" (UniqueName: \"kubernetes.io/projected/789ae719-5721-4edd-b05f-b5a312b3ec4c-kube-api-access-48frh\") pod \"payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd\" (UID: \"789ae719-5721-4edd-b05f-b5a312b3ec4c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd"
Sep 08 16:43:52.662975 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:52.662936    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f3c2c9-b701-4d0e-b5ce-047682111980-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt\" (UID: \"b0f3c2c9-b701-4d0e-b5ce-047682111980\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt"
Sep 08 16:43:52.663539 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:52.663519    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789ae719-5721-4edd-b05f-b5a312b3ec4c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd\" (UID: \"789ae719-5721-4edd-b05f-b5a312b3ec4c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd"
Sep 08 16:43:52.663615 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:52.663596    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f3c2c9-b701-4d0e-b5ce-047682111980-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt\" (UID: \"b0f3c2c9-b701-4d0e-b5ce-047682111980\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt"
Sep 08 16:43:52.671318 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:52.671282    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48frh\" (UniqueName: \"kubernetes.io/projected/789ae719-5721-4edd-b05f-b5a312b3ec4c-kube-api-access-48frh\") pod \"payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd\" (UID: \"789ae719-5721-4edd-b05f-b5a312b3ec4c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd"
Sep 08 16:43:52.671865 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:52.671806    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ql88\" (UniqueName: \"kubernetes.io/projected/b0f3c2c9-b701-4d0e-b5ce-047682111980-kube-api-access-4ql88\") pod \"payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt\" (UID: \"b0f3c2c9-b701-4d0e-b5ce-047682111980\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt"
Sep 08 16:43:52.751627 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:52.751550    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd"
Sep 08 16:43:52.818277 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:52.818248    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt"
Sep 08 16:43:53.032656 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:53.032628    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd"]
Sep 08 16:43:53.034315 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:43:53.034287    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789ae719_5721_4edd_b05f_b5a312b3ec4c.slice/crio-b94a20cb5f61a2ddda6d11c36d8383025f9e661eb38ac56c901e643040977c40 WatchSource:0}: Error finding container b94a20cb5f61a2ddda6d11c36d8383025f9e661eb38ac56c901e643040977c40: Status 404 returned error can't find the container with id b94a20cb5f61a2ddda6d11c36d8383025f9e661eb38ac56c901e643040977c40
Sep 08 16:43:53.099411 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:53.099387    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt"]
Sep 08 16:43:53.100510 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:43:53.100483    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f3c2c9_b701_4d0e_b5ce_047682111980.slice/crio-6a7bf5ae51e306377b60695fe92a5454cadba3bb540618ca497c626df7708fa4 WatchSource:0}: Error finding container 6a7bf5ae51e306377b60695fe92a5454cadba3bb540618ca497c626df7708fa4: Status 404 returned error can't find the container with id 6a7bf5ae51e306377b60695fe92a5454cadba3bb540618ca497c626df7708fa4
Sep 08 16:43:53.639456 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:53.639414    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd" event={"ID":"789ae719-5721-4edd-b05f-b5a312b3ec4c","Type":"ContainerStarted","Data":"e578d457afad87178a29bfd65e992cc46da691f6130e4001749310815f828932"}
Sep 08 16:43:53.639651 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:53.639478    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd" event={"ID":"789ae719-5721-4edd-b05f-b5a312b3ec4c","Type":"ContainerStarted","Data":"b94a20cb5f61a2ddda6d11c36d8383025f9e661eb38ac56c901e643040977c40"}
Sep 08 16:43:53.639651 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:53.639507    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd"
Sep 08 16:43:53.641277 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:53.641249    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt" event={"ID":"b0f3c2c9-b701-4d0e-b5ce-047682111980","Type":"ContainerStarted","Data":"f091aefdafff496abe0d6fc1516622f0ad9fcb4430e35feb6381e085da2760c7"}
Sep 08 16:43:53.641389 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:53.641291    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt" event={"ID":"b0f3c2c9-b701-4d0e-b5ce-047682111980","Type":"ContainerStarted","Data":"6a7bf5ae51e306377b60695fe92a5454cadba3bb540618ca497c626df7708fa4"}
Sep 08 16:43:53.641389 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:53.641358    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt"
Sep 08 16:43:53.661663 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:53.661609    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd" podStartSLOduration=1.661588626 podStartE2EDuration="1.661588626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:43:53.660239897 +0000 UTC m=+817.823211358" watchObservedRunningTime="2026-09-08 16:43:53.661588626 +0000 UTC m=+817.824560075"
Sep 08 16:43:53.682143 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:53.682091    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt" podStartSLOduration=1.682070097 podStartE2EDuration="1.682070097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:43:53.679023921 +0000 UTC m=+817.841995371" watchObservedRunningTime="2026-09-08 16:43:53.682070097 +0000 UTC m=+817.845041551"
Sep 08 16:43:55.287800 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:55.287764    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c81c932-wkcx5"]
Sep 08 16:43:55.291544 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:55.291528    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c81c932-wkcx5"
Sep 08 16:43:55.294438 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:55.294410    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:43:55.294547 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:55.294410    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-886jf\""
Sep 08 16:43:55.298673 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:55.298650    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c81c932-wkcx5"]
Sep 08 16:43:55.391222 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:55.391190    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64caa17c-ead1-4ef5-9a02-51c314fcbf08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4c81c932-wkcx5\" (UID: \"64caa17c-ead1-4ef5-9a02-51c314fcbf08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c81c932-wkcx5"
Sep 08 16:43:55.492636 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:55.492601    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64caa17c-ead1-4ef5-9a02-51c314fcbf08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4c81c932-wkcx5\" (UID: \"64caa17c-ead1-4ef5-9a02-51c314fcbf08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c81c932-wkcx5"
Sep 08 16:43:55.493235 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:55.493217    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64caa17c-ead1-4ef5-9a02-51c314fcbf08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4c81c932-wkcx5\" (UID: \"64caa17c-ead1-4ef5-9a02-51c314fcbf08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c81c932-wkcx5"
Sep 08 16:43:55.603624 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:55.603532    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c81c932-wkcx5"
Sep 08 16:43:55.878815 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:55.878792    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c81c932-wkcx5"]
Sep 08 16:43:55.880720 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:43:55.880690    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64caa17c_ead1_4ef5_9a02_51c314fcbf08.slice/crio-f6583c1bf5975dcd5587ae63493e7dd8583ea33f5eedfb7a21f12ee931e05c13 WatchSource:0}: Error finding container f6583c1bf5975dcd5587ae63493e7dd8583ea33f5eedfb7a21f12ee931e05c13: Status 404 returned error can't find the container with id f6583c1bf5975dcd5587ae63493e7dd8583ea33f5eedfb7a21f12ee931e05c13
Sep 08 16:43:56.657758 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:56.657727    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c81c932-wkcx5" event={"ID":"64caa17c-ead1-4ef5-9a02-51c314fcbf08","Type":"ContainerStarted","Data":"f6583c1bf5975dcd5587ae63493e7dd8583ea33f5eedfb7a21f12ee931e05c13"}
Sep 08 16:43:58.670641 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:58.670603    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c81c932-wkcx5" event={"ID":"64caa17c-ead1-4ef5-9a02-51c314fcbf08","Type":"ContainerStarted","Data":"3b9b792bef2117e137aa65cb96448243551d2a0cc35154022919b4a74b7a6f27"}
Sep 08 16:43:58.686367 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:58.686323    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c81c932-wkcx5" podStartSLOduration=1.312907106 podStartE2EDuration="3.68630801s" totalImagesPullingTime="2.373400904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:43:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:43:58.685209347 +0000 UTC m=+822.848180799" watchObservedRunningTime="2026-09-08 16:43:58.68630801 +0000 UTC m=+822.849279459"
Sep 08 16:43:59.677054 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:59.677019    2557 generic.go:356] "Generic (PLEG): container finished" podID="64caa17c-ead1-4ef5-9a02-51c314fcbf08" containerID="3b9b792bef2117e137aa65cb96448243551d2a0cc35154022919b4a74b7a6f27" exitCode=0
Sep 08 16:43:59.677402 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:43:59.677105    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c81c932-wkcx5" event={"ID":"64caa17c-ead1-4ef5-9a02-51c314fcbf08","Type":"ContainerDied","Data":"3b9b792bef2117e137aa65cb96448243551d2a0cc35154022919b4a74b7a6f27"}
Sep 08 16:44:00.817392 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:00.817366    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c81c932-wkcx5"
Sep 08 16:44:00.941212 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:00.941137    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64caa17c-ead1-4ef5-9a02-51c314fcbf08-maas-api-service-ca\") pod \"64caa17c-ead1-4ef5-9a02-51c314fcbf08\" (UID: \"64caa17c-ead1-4ef5-9a02-51c314fcbf08\") "
Sep 08 16:44:00.941467 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:00.941443    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64caa17c-ead1-4ef5-9a02-51c314fcbf08-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "64caa17c-ead1-4ef5-9a02-51c314fcbf08" (UID: "64caa17c-ead1-4ef5-9a02-51c314fcbf08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:44:01.042610 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:01.042570    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64caa17c-ead1-4ef5-9a02-51c314fcbf08-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:44:01.688144 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:01.688108    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c81c932-wkcx5" event={"ID":"64caa17c-ead1-4ef5-9a02-51c314fcbf08","Type":"ContainerDied","Data":"f6583c1bf5975dcd5587ae63493e7dd8583ea33f5eedfb7a21f12ee931e05c13"}
Sep 08 16:44:01.688318 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:01.688165    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6583c1bf5975dcd5587ae63493e7dd8583ea33f5eedfb7a21f12ee931e05c13"
Sep 08 16:44:01.688318 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:01.688120    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c81c932-wkcx5"
Sep 08 16:44:04.648764 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:04.648734    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd"
Sep 08 16:44:04.649228 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:04.648797    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt"
Sep 08 16:44:06.843612 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:06.843578    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:44:12.433192 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.433158    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8"]
Sep 08 16:44:12.433590 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.433396    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8" podUID="5beeb590-3aec-4c9f-9614-fd66209585f1" containerName="payload-processing" containerID="cri-o://e96d9707907e25ae2a1705e98cec59929d2c89d2bc17469b95667e013b6ffa02" gracePeriod=30
Sep 08 16:44:12.450426 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.450395    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v"]
Sep 08 16:44:12.450667 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.450631    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v" podUID="86683324-78e7-4949-89c6-92162a083ff3" containerName="payload-pre-processing" containerID="cri-o://c117c5fafda3135c9daa61dcda1a4d53c5cb060469213bd238bfd20068faccf0" gracePeriod=30
Sep 08 16:44:12.721408 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.721365    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c81c932-wkcx5"]
Sep 08 16:44:12.723274 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.723251    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c81c932-wkcx5"]
Sep 08 16:44:12.743185 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.743164    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8"
Sep 08 16:44:12.746057 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.746041    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v"
Sep 08 16:44:12.749069 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.749043    2557 generic.go:356] "Generic (PLEG): container finished" podID="5beeb590-3aec-4c9f-9614-fd66209585f1" containerID="e96d9707907e25ae2a1705e98cec59929d2c89d2bc17469b95667e013b6ffa02" exitCode=0
Sep 08 16:44:12.749164 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.749077    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8"
Sep 08 16:44:12.749164 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.749132    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8" event={"ID":"5beeb590-3aec-4c9f-9614-fd66209585f1","Type":"ContainerDied","Data":"e96d9707907e25ae2a1705e98cec59929d2c89d2bc17469b95667e013b6ffa02"}
Sep 08 16:44:12.749278 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.749192    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8" event={"ID":"5beeb590-3aec-4c9f-9614-fd66209585f1","Type":"ContainerDied","Data":"316e54ecb97a851282fb75a9e23e348e396cf85ac7154502becdfebad7fd7be6"}
Sep 08 16:44:12.749278 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.749226    2557 scope.go:117] "RemoveContainer" containerID="e96d9707907e25ae2a1705e98cec59929d2c89d2bc17469b95667e013b6ffa02"
Sep 08 16:44:12.750415 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.750397    2557 generic.go:356] "Generic (PLEG): container finished" podID="86683324-78e7-4949-89c6-92162a083ff3" containerID="c117c5fafda3135c9daa61dcda1a4d53c5cb060469213bd238bfd20068faccf0" exitCode=0
Sep 08 16:44:12.750491 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.750445    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v"
Sep 08 16:44:12.750491 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.750455    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v" event={"ID":"86683324-78e7-4949-89c6-92162a083ff3","Type":"ContainerDied","Data":"c117c5fafda3135c9daa61dcda1a4d53c5cb060469213bd238bfd20068faccf0"}
Sep 08 16:44:12.750562 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.750496    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v" event={"ID":"86683324-78e7-4949-89c6-92162a083ff3","Type":"ContainerDied","Data":"bfb569da23f3b41f3ccc21f3f33fab99b1fb238e47af3618371d2ac2ca337bb6"}
Sep 08 16:44:12.757466 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.757451    2557 scope.go:117] "RemoveContainer" containerID="e96d9707907e25ae2a1705e98cec59929d2c89d2bc17469b95667e013b6ffa02"
Sep 08 16:44:12.757724 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:44:12.757706    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e96d9707907e25ae2a1705e98cec59929d2c89d2bc17469b95667e013b6ffa02\": container with ID starting with e96d9707907e25ae2a1705e98cec59929d2c89d2bc17469b95667e013b6ffa02 not found: ID does not exist" containerID="e96d9707907e25ae2a1705e98cec59929d2c89d2bc17469b95667e013b6ffa02"
Sep 08 16:44:12.757777 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.757733    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e96d9707907e25ae2a1705e98cec59929d2c89d2bc17469b95667e013b6ffa02"} err="failed to get container status \"e96d9707907e25ae2a1705e98cec59929d2c89d2bc17469b95667e013b6ffa02\": rpc error: code = NotFound desc = could not find container \"e96d9707907e25ae2a1705e98cec59929d2c89d2bc17469b95667e013b6ffa02\": container with ID starting with e96d9707907e25ae2a1705e98cec59929d2c89d2bc17469b95667e013b6ffa02 not found: ID does not exist"
Sep 08 16:44:12.757777 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.757751    2557 scope.go:117] "RemoveContainer" containerID="c117c5fafda3135c9daa61dcda1a4d53c5cb060469213bd238bfd20068faccf0"
Sep 08 16:44:12.767732 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.767076    2557 scope.go:117] "RemoveContainer" containerID="c117c5fafda3135c9daa61dcda1a4d53c5cb060469213bd238bfd20068faccf0"
Sep 08 16:44:12.767732 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:44:12.767426    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c117c5fafda3135c9daa61dcda1a4d53c5cb060469213bd238bfd20068faccf0\": container with ID starting with c117c5fafda3135c9daa61dcda1a4d53c5cb060469213bd238bfd20068faccf0 not found: ID does not exist" containerID="c117c5fafda3135c9daa61dcda1a4d53c5cb060469213bd238bfd20068faccf0"
Sep 08 16:44:12.767732 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.767449    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c117c5fafda3135c9daa61dcda1a4d53c5cb060469213bd238bfd20068faccf0"} err="failed to get container status \"c117c5fafda3135c9daa61dcda1a4d53c5cb060469213bd238bfd20068faccf0\": rpc error: code = NotFound desc = could not find container \"c117c5fafda3135c9daa61dcda1a4d53c5cb060469213bd238bfd20068faccf0\": container with ID starting with c117c5fafda3135c9daa61dcda1a4d53c5cb060469213bd238bfd20068faccf0 not found: ID does not exist"
Sep 08 16:44:12.848156 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.848125    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq7pn\" (UniqueName: \"kubernetes.io/projected/5beeb590-3aec-4c9f-9614-fd66209585f1-kube-api-access-dq7pn\") pod \"5beeb590-3aec-4c9f-9614-fd66209585f1\" (UID: \"5beeb590-3aec-4c9f-9614-fd66209585f1\") "
Sep 08 16:44:12.848328 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.848171    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5beeb590-3aec-4c9f-9614-fd66209585f1-plugins-config-volume\") pod \"5beeb590-3aec-4c9f-9614-fd66209585f1\" (UID: \"5beeb590-3aec-4c9f-9614-fd66209585f1\") "
Sep 08 16:44:12.848328 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.848192    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86683324-78e7-4949-89c6-92162a083ff3-plugins-config-volume\") pod \"86683324-78e7-4949-89c6-92162a083ff3\" (UID: \"86683324-78e7-4949-89c6-92162a083ff3\") "
Sep 08 16:44:12.848328 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.848235    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcdtr\" (UniqueName: \"kubernetes.io/projected/86683324-78e7-4949-89c6-92162a083ff3-kube-api-access-tcdtr\") pod \"86683324-78e7-4949-89c6-92162a083ff3\" (UID: \"86683324-78e7-4949-89c6-92162a083ff3\") "
Sep 08 16:44:12.850273 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.848816    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5beeb590-3aec-4c9f-9614-fd66209585f1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5beeb590-3aec-4c9f-9614-fd66209585f1" (UID: "5beeb590-3aec-4c9f-9614-fd66209585f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:44:12.854864 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.852924    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86683324-78e7-4949-89c6-92162a083ff3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "86683324-78e7-4949-89c6-92162a083ff3" (UID: "86683324-78e7-4949-89c6-92162a083ff3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:44:12.857445 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.857416    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86683324-78e7-4949-89c6-92162a083ff3-kube-api-access-tcdtr" (OuterVolumeSpecName: "kube-api-access-tcdtr") pod "86683324-78e7-4949-89c6-92162a083ff3" (UID: "86683324-78e7-4949-89c6-92162a083ff3"). InnerVolumeSpecName "kube-api-access-tcdtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:44:12.857574 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.857550    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5beeb590-3aec-4c9f-9614-fd66209585f1-kube-api-access-dq7pn" (OuterVolumeSpecName: "kube-api-access-dq7pn") pod "5beeb590-3aec-4c9f-9614-fd66209585f1" (UID: "5beeb590-3aec-4c9f-9614-fd66209585f1"). InnerVolumeSpecName "kube-api-access-dq7pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:44:12.949884 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.949805    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq7pn\" (UniqueName: \"kubernetes.io/projected/5beeb590-3aec-4c9f-9614-fd66209585f1-kube-api-access-dq7pn\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:44:12.949884 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.949849    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5beeb590-3aec-4c9f-9614-fd66209585f1-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:44:12.949884 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.949859    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86683324-78e7-4949-89c6-92162a083ff3-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:44:12.949884 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:12.949868    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcdtr\" (UniqueName: \"kubernetes.io/projected/86683324-78e7-4949-89c6-92162a083ff3-kube-api-access-tcdtr\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:44:13.074488 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:13.074452    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v"]
Sep 08 16:44:13.077716 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:13.077694    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c81c932-66b9555b9d-skg5v"]
Sep 08 16:44:13.092335 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:13.092309    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8"]
Sep 08 16:44:13.096477 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:13.096455    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c81c932-5d99bbb9fc-qlss8"]
Sep 08 16:44:14.345091 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:14.345063    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5beeb590-3aec-4c9f-9614-fd66209585f1" path="/var/lib/kubelet/pods/5beeb590-3aec-4c9f-9614-fd66209585f1/volumes"
Sep 08 16:44:14.345433 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:14.345370    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64caa17c-ead1-4ef5-9a02-51c314fcbf08" path="/var/lib/kubelet/pods/64caa17c-ead1-4ef5-9a02-51c314fcbf08/volumes"
Sep 08 16:44:14.345656 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:14.345643    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86683324-78e7-4949-89c6-92162a083ff3" path="/var/lib/kubelet/pods/86683324-78e7-4949-89c6-92162a083ff3/volumes"
Sep 08 16:44:17.356537 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.356505    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq"]
Sep 08 16:44:17.356901 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.356861    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86683324-78e7-4949-89c6-92162a083ff3" containerName="payload-pre-processing"
Sep 08 16:44:17.356901 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.356873    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="86683324-78e7-4949-89c6-92162a083ff3" containerName="payload-pre-processing"
Sep 08 16:44:17.356901 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.356888    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64caa17c-ead1-4ef5-9a02-51c314fcbf08" containerName="revoke-keys"
Sep 08 16:44:17.356901 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.356894    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="64caa17c-ead1-4ef5-9a02-51c314fcbf08" containerName="revoke-keys"
Sep 08 16:44:17.357034 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.356904    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5beeb590-3aec-4c9f-9614-fd66209585f1" containerName="payload-processing"
Sep 08 16:44:17.357034 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.356912    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5beeb590-3aec-4c9f-9614-fd66209585f1" containerName="payload-processing"
Sep 08 16:44:17.357034 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.356965    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="64caa17c-ead1-4ef5-9a02-51c314fcbf08" containerName="revoke-keys"
Sep 08 16:44:17.357034 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.356974    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5beeb590-3aec-4c9f-9614-fd66209585f1" containerName="payload-processing"
Sep 08 16:44:17.357034 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.356980    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="86683324-78e7-4949-89c6-92162a083ff3" containerName="payload-pre-processing"
Sep 08 16:44:17.361624 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.361604    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq"
Sep 08 16:44:17.364450 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.364430    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-fc1daf8b\""
Sep 08 16:44:17.364567 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.364433    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-fc1daf8b-dockercfg-zffwh\""
Sep 08 16:44:17.373876 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.373850    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq"]
Sep 08 16:44:17.444275 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.444244    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq"]
Sep 08 16:44:17.447518 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.447504    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq"
Sep 08 16:44:17.462578 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.462556    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq"]
Sep 08 16:44:17.494856 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.490277    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnb8t\" (UniqueName: \"kubernetes.io/projected/6c8cfdd9-5bc3-4ef5-8993-4999d232a24f-kube-api-access-mnb8t\") pod \"payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq\" (UID: \"6c8cfdd9-5bc3-4ef5-8993-4999d232a24f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq"
Sep 08 16:44:17.494856 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.490331    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8cfdd9-5bc3-4ef5-8993-4999d232a24f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq\" (UID: \"6c8cfdd9-5bc3-4ef5-8993-4999d232a24f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq"
Sep 08 16:44:17.591402 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.591366    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnb8t\" (UniqueName: \"kubernetes.io/projected/6c8cfdd9-5bc3-4ef5-8993-4999d232a24f-kube-api-access-mnb8t\") pod \"payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq\" (UID: \"6c8cfdd9-5bc3-4ef5-8993-4999d232a24f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq"
Sep 08 16:44:17.591402 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.591407    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8cfdd9-5bc3-4ef5-8993-4999d232a24f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq\" (UID: \"6c8cfdd9-5bc3-4ef5-8993-4999d232a24f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq"
Sep 08 16:44:17.591643 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.591442    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b02cd05-e759-4931-a47d-ea313d6bbb87-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq\" (UID: \"4b02cd05-e759-4931-a47d-ea313d6bbb87\") " pod="openshift-ingress/payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq"
Sep 08 16:44:17.591643 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.591467    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c66k\" (UniqueName: \"kubernetes.io/projected/4b02cd05-e759-4931-a47d-ea313d6bbb87-kube-api-access-5c66k\") pod \"payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq\" (UID: \"4b02cd05-e759-4931-a47d-ea313d6bbb87\") " pod="openshift-ingress/payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq"
Sep 08 16:44:17.592141 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.592121    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8cfdd9-5bc3-4ef5-8993-4999d232a24f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq\" (UID: \"6c8cfdd9-5bc3-4ef5-8993-4999d232a24f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq"
Sep 08 16:44:17.600746 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.600721    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnb8t\" (UniqueName: \"kubernetes.io/projected/6c8cfdd9-5bc3-4ef5-8993-4999d232a24f-kube-api-access-mnb8t\") pod \"payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq\" (UID: \"6c8cfdd9-5bc3-4ef5-8993-4999d232a24f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq"
Sep 08 16:44:17.671587 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.671557    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq"
Sep 08 16:44:17.692329 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.692297    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b02cd05-e759-4931-a47d-ea313d6bbb87-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq\" (UID: \"4b02cd05-e759-4931-a47d-ea313d6bbb87\") " pod="openshift-ingress/payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq"
Sep 08 16:44:17.692430 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.692358    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c66k\" (UniqueName: \"kubernetes.io/projected/4b02cd05-e759-4931-a47d-ea313d6bbb87-kube-api-access-5c66k\") pod \"payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq\" (UID: \"4b02cd05-e759-4931-a47d-ea313d6bbb87\") " pod="openshift-ingress/payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq"
Sep 08 16:44:17.692983 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.692966    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b02cd05-e759-4931-a47d-ea313d6bbb87-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq\" (UID: \"4b02cd05-e759-4931-a47d-ea313d6bbb87\") " pod="openshift-ingress/payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq"
Sep 08 16:44:17.702935 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.702912    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c66k\" (UniqueName: \"kubernetes.io/projected/4b02cd05-e759-4931-a47d-ea313d6bbb87-kube-api-access-5c66k\") pod \"payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq\" (UID: \"4b02cd05-e759-4931-a47d-ea313d6bbb87\") " pod="openshift-ingress/payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq"
Sep 08 16:44:17.757348 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.757313    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq"
Sep 08 16:44:17.953343 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:17.953216    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq"]
Sep 08 16:44:17.955451 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:44:17.955424    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8cfdd9_5bc3_4ef5_8993_4999d232a24f.slice/crio-5c37993e11d1742d03dd1e0b73cca9fda17458aa88e7c208749c5ab8b25110c7 WatchSource:0}: Error finding container 5c37993e11d1742d03dd1e0b73cca9fda17458aa88e7c208749c5ab8b25110c7: Status 404 returned error can't find the container with id 5c37993e11d1742d03dd1e0b73cca9fda17458aa88e7c208749c5ab8b25110c7
Sep 08 16:44:18.037518 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:18.037495    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq"]
Sep 08 16:44:18.040005 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:44:18.039975    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b02cd05_e759_4931_a47d_ea313d6bbb87.slice/crio-0b2743a5a5788e91192a75802faa7715dbc32176f2dac2d908c1990fb1e9d9e4 WatchSource:0}: Error finding container 0b2743a5a5788e91192a75802faa7715dbc32176f2dac2d908c1990fb1e9d9e4: Status 404 returned error can't find the container with id 0b2743a5a5788e91192a75802faa7715dbc32176f2dac2d908c1990fb1e9d9e4
Sep 08 16:44:18.785607 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:18.785576    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq" event={"ID":"6c8cfdd9-5bc3-4ef5-8993-4999d232a24f","Type":"ContainerStarted","Data":"b98739d40ed453d11d8e3349c1a5eeafd25ebca5cbf6784d5e4b36c0ff13037b"}
Sep 08 16:44:18.786050 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:18.785624    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq" event={"ID":"6c8cfdd9-5bc3-4ef5-8993-4999d232a24f","Type":"ContainerStarted","Data":"5c37993e11d1742d03dd1e0b73cca9fda17458aa88e7c208749c5ab8b25110c7"}
Sep 08 16:44:18.786050 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:18.785691    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq"
Sep 08 16:44:18.787447 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:18.787423    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq" event={"ID":"4b02cd05-e759-4931-a47d-ea313d6bbb87","Type":"ContainerStarted","Data":"77a594bac5526ac325200de35641f2753d6f1c23db22619b83cda2e1a1ca871f"}
Sep 08 16:44:18.787573 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:18.787460    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq" event={"ID":"4b02cd05-e759-4931-a47d-ea313d6bbb87","Type":"ContainerStarted","Data":"0b2743a5a5788e91192a75802faa7715dbc32176f2dac2d908c1990fb1e9d9e4"}
Sep 08 16:44:18.787573 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:18.787517    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq"
Sep 08 16:44:18.805179 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:18.805135    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq" podStartSLOduration=1.805121921 podStartE2EDuration="1.805121921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:44:18.804275496 +0000 UTC m=+842.967246944" watchObservedRunningTime="2026-09-08 16:44:18.805121921 +0000 UTC m=+842.968093368"
Sep 08 16:44:18.823359 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:18.823318    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq" podStartSLOduration=1.823304646 podStartE2EDuration="1.823304646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:44:18.821363879 +0000 UTC m=+842.984335352" watchObservedRunningTime="2026-09-08 16:44:18.823304646 +0000 UTC m=+842.986276094"
Sep 08 16:44:22.648599 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:22.648566    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:44:25.418486 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:25.418451    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"]
Sep 08 16:44:25.422297 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:25.422278    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:44:25.425073 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:25.425053    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-1be25697-kserve-self-signed-certs\""
Sep 08 16:44:25.431074 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:25.431053    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"]
Sep 08 16:44:25.464429 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:25.464398    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e58b226-2f45-46a0-ad65-f0eaebd08019-tmp-dir\") pod \"e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf\" (UID: \"5e58b226-2f45-46a0-ad65-f0eaebd08019\") " pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:44:25.464601 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:25.464450    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e58b226-2f45-46a0-ad65-f0eaebd08019-home\") pod \"e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf\" (UID: \"5e58b226-2f45-46a0-ad65-f0eaebd08019\") " pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:44:25.464601 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:25.464512    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e58b226-2f45-46a0-ad65-f0eaebd08019-dshm\") pod \"e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf\" (UID: \"5e58b226-2f45-46a0-ad65-f0eaebd08019\") " pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:44:25.464601 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:25.464538    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e58b226-2f45-46a0-ad65-f0eaebd08019-tls-certs\") pod \"e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf\" (UID: \"5e58b226-2f45-46a0-ad65-f0eaebd08019\") " pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:44:25.464601 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:25.464575    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmjgg\" (UniqueName: \"kubernetes.io/projected/5e58b226-2f45-46a0-ad65-f0eaebd08019-kube-api-access-xmjgg\") pod \"e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf\" (UID: \"5e58b226-2f45-46a0-ad65-f0eaebd08019\") " pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:44:25.464601 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:25.464601    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e58b226-2f45-46a0-ad65-f0eaebd08019-model-cache\") pod \"e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf\" (UID: \"5e58b226-2f45-46a0-ad65-f0eaebd08019\") " pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:44:25.565150 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:25.565113    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e58b226-2f45-46a0-ad65-f0eaebd08019-dshm\") pod \"e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf\" (UID: \"5e58b226-2f45-46a0-ad65-f0eaebd08019\") " pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:44:25.565150 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:25.565151    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e58b226-2f45-46a0-ad65-f0eaebd08019-tls-certs\") pod \"e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf\" (UID: \"5e58b226-2f45-46a0-ad65-f0eaebd08019\") " pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:44:25.565372 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:25.565172    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmjgg\" (UniqueName: \"kubernetes.io/projected/5e58b226-2f45-46a0-ad65-f0eaebd08019-kube-api-access-xmjgg\") pod \"e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf\" (UID: \"5e58b226-2f45-46a0-ad65-f0eaebd08019\") " pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:44:25.565372 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:25.565241    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e58b226-2f45-46a0-ad65-f0eaebd08019-model-cache\") pod \"e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf\" (UID: \"5e58b226-2f45-46a0-ad65-f0eaebd08019\") " pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:44:25.565372 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:25.565291    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e58b226-2f45-46a0-ad65-f0eaebd08019-tmp-dir\") pod \"e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf\" (UID: \"5e58b226-2f45-46a0-ad65-f0eaebd08019\") " pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:44:25.565372 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:25.565357    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e58b226-2f45-46a0-ad65-f0eaebd08019-home\") pod \"e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf\" (UID: \"5e58b226-2f45-46a0-ad65-f0eaebd08019\") " pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:44:25.565669 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:25.565643    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e58b226-2f45-46a0-ad65-f0eaebd08019-model-cache\") pod \"e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf\" (UID: \"5e58b226-2f45-46a0-ad65-f0eaebd08019\") " pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:44:25.565734 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:25.565664    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e58b226-2f45-46a0-ad65-f0eaebd08019-tmp-dir\") pod \"e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf\" (UID: \"5e58b226-2f45-46a0-ad65-f0eaebd08019\") " pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:44:25.565734 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:25.565691    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e58b226-2f45-46a0-ad65-f0eaebd08019-home\") pod \"e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf\" (UID: \"5e58b226-2f45-46a0-ad65-f0eaebd08019\") " pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:44:25.567445 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:25.567429    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e58b226-2f45-46a0-ad65-f0eaebd08019-dshm\") pod \"e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf\" (UID: \"5e58b226-2f45-46a0-ad65-f0eaebd08019\") " pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:44:25.567766 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:25.567746    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e58b226-2f45-46a0-ad65-f0eaebd08019-tls-certs\") pod \"e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf\" (UID: \"5e58b226-2f45-46a0-ad65-f0eaebd08019\") " pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:44:25.574083 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:25.574057    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmjgg\" (UniqueName: \"kubernetes.io/projected/5e58b226-2f45-46a0-ad65-f0eaebd08019-kube-api-access-xmjgg\") pod \"e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf\" (UID: \"5e58b226-2f45-46a0-ad65-f0eaebd08019\") " pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:44:25.732234 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:25.732150    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:44:26.004068 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:26.004043    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"]
Sep 08 16:44:26.006760 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:44:26.006732    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e58b226_2f45_46a0_ad65_f0eaebd08019.slice/crio-fefc44480610d1b1b15b063bc42df3460ee84f711198b1989c6a0a9dbe137e06 WatchSource:0}: Error finding container fefc44480610d1b1b15b063bc42df3460ee84f711198b1989c6a0a9dbe137e06: Status 404 returned error can't find the container with id fefc44480610d1b1b15b063bc42df3460ee84f711198b1989c6a0a9dbe137e06
Sep 08 16:44:26.830002 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:26.829961    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf" event={"ID":"5e58b226-2f45-46a0-ad65-f0eaebd08019","Type":"ContainerStarted","Data":"734fb623b3109ad74099abef326eafdc194a5b80aff558aa7ef8cfacdddf589d"}
Sep 08 16:44:26.830465 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:26.830014    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf" event={"ID":"5e58b226-2f45-46a0-ad65-f0eaebd08019","Type":"ContainerStarted","Data":"fefc44480610d1b1b15b063bc42df3460ee84f711198b1989c6a0a9dbe137e06"}
Sep 08 16:44:26.847953 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:26.847897    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf" podStartSLOduration=1.847881017 podStartE2EDuration="1.847881017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:44:26.846051075 +0000 UTC m=+851.009022663" watchObservedRunningTime="2026-09-08 16:44:26.847881017 +0000 UTC m=+851.010852502"
Sep 08 16:44:29.795480 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:29.795451    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq"
Sep 08 16:44:29.795892 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:29.795522    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq"
Sep 08 16:44:35.732456 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:35.732412    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:44:35.732965 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:35.732468    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:44:35.745524 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:35.745501    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:44:35.886271 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:35.886240    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:44:38.862757 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:38.862728    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:44:41.195550 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:41.195521    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1daf8b-hgr9j"]
Sep 08 16:44:41.197846 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:41.197821    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1daf8b-hgr9j"
Sep 08 16:44:41.200436 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:41.200415    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-886jf\""
Sep 08 16:44:41.200563 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:41.200415    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:44:41.209304 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:41.209281    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1daf8b-hgr9j"]
Sep 08 16:44:41.300929 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:41.300900    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbcffed-b2da-434a-904d-f8197579b7f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fc1daf8b-hgr9j\" (UID: \"0cbcffed-b2da-434a-904d-f8197579b7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1daf8b-hgr9j"
Sep 08 16:44:41.402035 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:41.402006    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbcffed-b2da-434a-904d-f8197579b7f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fc1daf8b-hgr9j\" (UID: \"0cbcffed-b2da-434a-904d-f8197579b7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1daf8b-hgr9j"
Sep 08 16:44:41.402622 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:41.402604    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbcffed-b2da-434a-904d-f8197579b7f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fc1daf8b-hgr9j\" (UID: \"0cbcffed-b2da-434a-904d-f8197579b7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1daf8b-hgr9j"
Sep 08 16:44:41.509802 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:41.509739    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1daf8b-hgr9j"
Sep 08 16:44:41.779326 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:41.779240    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1daf8b-hgr9j"]
Sep 08 16:44:41.783821 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:44:41.783794    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbcffed_b2da_434a_904d_f8197579b7f1.slice/crio-c08f92537a11478bbf89eecfea599c20b73e438f9299879aeae4a46dfd45fd18 WatchSource:0}: Error finding container c08f92537a11478bbf89eecfea599c20b73e438f9299879aeae4a46dfd45fd18: Status 404 returned error can't find the container with id c08f92537a11478bbf89eecfea599c20b73e438f9299879aeae4a46dfd45fd18
Sep 08 16:44:41.904421 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:41.904391    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1daf8b-hgr9j" event={"ID":"0cbcffed-b2da-434a-904d-f8197579b7f1","Type":"ContainerStarted","Data":"c08f92537a11478bbf89eecfea599c20b73e438f9299879aeae4a46dfd45fd18"}
Sep 08 16:44:42.910996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:42.910963    2557 generic.go:356] "Generic (PLEG): container finished" podID="0cbcffed-b2da-434a-904d-f8197579b7f1" containerID="70c4ac6dedec8841ab738c76445434f26ca457068da7088f52ed9b5182ab7229" exitCode=0
Sep 08 16:44:42.911395 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:42.911043    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1daf8b-hgr9j" event={"ID":"0cbcffed-b2da-434a-904d-f8197579b7f1","Type":"ContainerDied","Data":"70c4ac6dedec8841ab738c76445434f26ca457068da7088f52ed9b5182ab7229"}
Sep 08 16:44:44.048160 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:44.048135    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1daf8b-hgr9j"
Sep 08 16:44:44.122305 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:44.122275    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbcffed-b2da-434a-904d-f8197579b7f1-maas-api-service-ca\") pod \"0cbcffed-b2da-434a-904d-f8197579b7f1\" (UID: \"0cbcffed-b2da-434a-904d-f8197579b7f1\") "
Sep 08 16:44:44.122635 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:44.122612    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cbcffed-b2da-434a-904d-f8197579b7f1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0cbcffed-b2da-434a-904d-f8197579b7f1" (UID: "0cbcffed-b2da-434a-904d-f8197579b7f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:44:44.222900 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:44.222819    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbcffed-b2da-434a-904d-f8197579b7f1-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:44:44.922735 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:44.922709    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1daf8b-hgr9j"
Sep 08 16:44:44.922933 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:44.922708    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1daf8b-hgr9j" event={"ID":"0cbcffed-b2da-434a-904d-f8197579b7f1","Type":"ContainerDied","Data":"c08f92537a11478bbf89eecfea599c20b73e438f9299879aeae4a46dfd45fd18"}
Sep 08 16:44:44.922933 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:44.922859    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c08f92537a11478bbf89eecfea599c20b73e438f9299879aeae4a46dfd45fd18"
Sep 08 16:44:46.637358 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.637325    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"]
Sep 08 16:44:46.637730 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.637695    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cbcffed-b2da-434a-904d-f8197579b7f1" containerName="revoke-keys"
Sep 08 16:44:46.637730 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.637705    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cbcffed-b2da-434a-904d-f8197579b7f1" containerName="revoke-keys"
Sep 08 16:44:46.637803 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.637763    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cbcffed-b2da-434a-904d-f8197579b7f1" containerName="revoke-keys"
Sep 08 16:44:46.639754 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.639736    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:44:46.642320 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.642301    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-4cea93\"/\"default-dockercfg-vnh2q\""
Sep 08 16:44:46.642438 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.642300    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-4cea93\"/\"openshift-service-ca.crt\""
Sep 08 16:44:46.643327 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.643314    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-4cea93\"/\"kube-root-ca.crt\""
Sep 08 16:44:46.643369 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.643351    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-4cea93\"/\"autores-model-4cea93-kserve-self-signed-certs\""
Sep 08 16:44:46.653099 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.653076    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"]
Sep 08 16:44:46.743208 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.743177    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrzrl\" (UniqueName: \"kubernetes.io/projected/06040053-0109-456a-8c47-a7fdc2711b70-kube-api-access-rrzrl\") pod \"autores-model-4cea93-kserve-75c49ccfc5-xspss\" (UID: \"06040053-0109-456a-8c47-a7fdc2711b70\") " pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:44:46.743349 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.743215    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06040053-0109-456a-8c47-a7fdc2711b70-model-cache\") pod \"autores-model-4cea93-kserve-75c49ccfc5-xspss\" (UID: \"06040053-0109-456a-8c47-a7fdc2711b70\") " pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:44:46.743349 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.743298    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06040053-0109-456a-8c47-a7fdc2711b70-tmp-dir\") pod \"autores-model-4cea93-kserve-75c49ccfc5-xspss\" (UID: \"06040053-0109-456a-8c47-a7fdc2711b70\") " pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:44:46.743349 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.743335    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06040053-0109-456a-8c47-a7fdc2711b70-dshm\") pod \"autores-model-4cea93-kserve-75c49ccfc5-xspss\" (UID: \"06040053-0109-456a-8c47-a7fdc2711b70\") " pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:44:46.743454 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.743378    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06040053-0109-456a-8c47-a7fdc2711b70-tls-certs\") pod \"autores-model-4cea93-kserve-75c49ccfc5-xspss\" (UID: \"06040053-0109-456a-8c47-a7fdc2711b70\") " pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:44:46.743454 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.743437    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06040053-0109-456a-8c47-a7fdc2711b70-home\") pod \"autores-model-4cea93-kserve-75c49ccfc5-xspss\" (UID: \"06040053-0109-456a-8c47-a7fdc2711b70\") " pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:44:46.844634 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.844603    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06040053-0109-456a-8c47-a7fdc2711b70-tmp-dir\") pod \"autores-model-4cea93-kserve-75c49ccfc5-xspss\" (UID: \"06040053-0109-456a-8c47-a7fdc2711b70\") " pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:44:46.844634 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.844642    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06040053-0109-456a-8c47-a7fdc2711b70-dshm\") pod \"autores-model-4cea93-kserve-75c49ccfc5-xspss\" (UID: \"06040053-0109-456a-8c47-a7fdc2711b70\") " pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:44:46.844904 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.844664    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06040053-0109-456a-8c47-a7fdc2711b70-tls-certs\") pod \"autores-model-4cea93-kserve-75c49ccfc5-xspss\" (UID: \"06040053-0109-456a-8c47-a7fdc2711b70\") " pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:44:46.844904 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.844699    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06040053-0109-456a-8c47-a7fdc2711b70-home\") pod \"autores-model-4cea93-kserve-75c49ccfc5-xspss\" (UID: \"06040053-0109-456a-8c47-a7fdc2711b70\") " pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:44:46.844904 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.844771    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrzrl\" (UniqueName: \"kubernetes.io/projected/06040053-0109-456a-8c47-a7fdc2711b70-kube-api-access-rrzrl\") pod \"autores-model-4cea93-kserve-75c49ccfc5-xspss\" (UID: \"06040053-0109-456a-8c47-a7fdc2711b70\") " pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:44:46.844904 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.844801    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06040053-0109-456a-8c47-a7fdc2711b70-model-cache\") pod \"autores-model-4cea93-kserve-75c49ccfc5-xspss\" (UID: \"06040053-0109-456a-8c47-a7fdc2711b70\") " pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:44:46.845120 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.845089    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06040053-0109-456a-8c47-a7fdc2711b70-tmp-dir\") pod \"autores-model-4cea93-kserve-75c49ccfc5-xspss\" (UID: \"06040053-0109-456a-8c47-a7fdc2711b70\") " pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:44:46.845178 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.845159    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06040053-0109-456a-8c47-a7fdc2711b70-model-cache\") pod \"autores-model-4cea93-kserve-75c49ccfc5-xspss\" (UID: \"06040053-0109-456a-8c47-a7fdc2711b70\") " pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:44:46.845234 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.845208    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06040053-0109-456a-8c47-a7fdc2711b70-home\") pod \"autores-model-4cea93-kserve-75c49ccfc5-xspss\" (UID: \"06040053-0109-456a-8c47-a7fdc2711b70\") " pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:44:46.847200 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.847171    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06040053-0109-456a-8c47-a7fdc2711b70-dshm\") pod \"autores-model-4cea93-kserve-75c49ccfc5-xspss\" (UID: \"06040053-0109-456a-8c47-a7fdc2711b70\") " pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:44:46.847303 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.847259    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06040053-0109-456a-8c47-a7fdc2711b70-tls-certs\") pod \"autores-model-4cea93-kserve-75c49ccfc5-xspss\" (UID: \"06040053-0109-456a-8c47-a7fdc2711b70\") " pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:44:46.854142 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.854117    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrzrl\" (UniqueName: \"kubernetes.io/projected/06040053-0109-456a-8c47-a7fdc2711b70-kube-api-access-rrzrl\") pod \"autores-model-4cea93-kserve-75c49ccfc5-xspss\" (UID: \"06040053-0109-456a-8c47-a7fdc2711b70\") " pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:44:46.948974 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:46.948945    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:44:47.248776 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:47.248709    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"]
Sep 08 16:44:47.251844 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:44:47.251801    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06040053_0109_456a_8c47_a7fdc2711b70.slice/crio-74179e253da68501df30f567434f8b81e6249bfd6be4db70362ce629882e353d WatchSource:0}: Error finding container 74179e253da68501df30f567434f8b81e6249bfd6be4db70362ce629882e353d: Status 404 returned error can't find the container with id 74179e253da68501df30f567434f8b81e6249bfd6be4db70362ce629882e353d
Sep 08 16:44:47.939355 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:47.938788    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss" event={"ID":"06040053-0109-456a-8c47-a7fdc2711b70","Type":"ContainerStarted","Data":"590b4a25802b6d1c3a47b476f7559870a0e2dd8d98287c038af3d066409db95f"}
Sep 08 16:44:47.939355 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:47.938855    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss" event={"ID":"06040053-0109-456a-8c47-a7fdc2711b70","Type":"ContainerStarted","Data":"74179e253da68501df30f567434f8b81e6249bfd6be4db70362ce629882e353d"}
Sep 08 16:44:48.042578 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.042530    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss" podStartSLOduration=2.042512898 podStartE2EDuration="2.042512898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:44:47.967799329 +0000 UTC m=+872.130770777" watchObservedRunningTime="2026-09-08 16:44:48.042512898 +0000 UTC m=+872.205484346"
Sep 08 16:44:48.044202 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.044179    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq"]
Sep 08 16:44:48.044389 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.044361    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq" podUID="4b02cd05-e759-4931-a47d-ea313d6bbb87" containerName="payload-processing" containerID="cri-o://77a594bac5526ac325200de35641f2753d6f1c23db22619b83cda2e1a1ca871f" gracePeriod=30
Sep 08 16:44:48.053563 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.052932    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq"]
Sep 08 16:44:48.053907 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.053189    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq" podUID="6c8cfdd9-5bc3-4ef5-8993-4999d232a24f" containerName="payload-pre-processing" containerID="cri-o://b98739d40ed453d11d8e3349c1a5eeafd25ebca5cbf6784d5e4b36c0ff13037b" gracePeriod=30
Sep 08 16:44:48.278748 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.278717    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1daf8b-hgr9j"]
Sep 08 16:44:48.283192 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.283165    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1daf8b-hgr9j"]
Sep 08 16:44:48.317741 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.317720    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq"
Sep 08 16:44:48.320982 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.320961    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq"
Sep 08 16:44:48.347343 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.347318    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cbcffed-b2da-434a-904d-f8197579b7f1" path="/var/lib/kubelet/pods/0cbcffed-b2da-434a-904d-f8197579b7f1/volumes"
Sep 08 16:44:48.461050 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.461023    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnb8t\" (UniqueName: \"kubernetes.io/projected/6c8cfdd9-5bc3-4ef5-8993-4999d232a24f-kube-api-access-mnb8t\") pod \"6c8cfdd9-5bc3-4ef5-8993-4999d232a24f\" (UID: \"6c8cfdd9-5bc3-4ef5-8993-4999d232a24f\") "
Sep 08 16:44:48.461217 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.461072    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5c66k\" (UniqueName: \"kubernetes.io/projected/4b02cd05-e759-4931-a47d-ea313d6bbb87-kube-api-access-5c66k\") pod \"4b02cd05-e759-4931-a47d-ea313d6bbb87\" (UID: \"4b02cd05-e759-4931-a47d-ea313d6bbb87\") "
Sep 08 16:44:48.461217 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.461102    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8cfdd9-5bc3-4ef5-8993-4999d232a24f-plugins-config-volume\") pod \"6c8cfdd9-5bc3-4ef5-8993-4999d232a24f\" (UID: \"6c8cfdd9-5bc3-4ef5-8993-4999d232a24f\") "
Sep 08 16:44:48.461217 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.461131    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b02cd05-e759-4931-a47d-ea313d6bbb87-plugins-config-volume\") pod \"4b02cd05-e759-4931-a47d-ea313d6bbb87\" (UID: \"4b02cd05-e759-4931-a47d-ea313d6bbb87\") "
Sep 08 16:44:48.461535 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.461497    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c8cfdd9-5bc3-4ef5-8993-4999d232a24f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6c8cfdd9-5bc3-4ef5-8993-4999d232a24f" (UID: "6c8cfdd9-5bc3-4ef5-8993-4999d232a24f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:44:48.461640 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.461548    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b02cd05-e759-4931-a47d-ea313d6bbb87-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b02cd05-e759-4931-a47d-ea313d6bbb87" (UID: "4b02cd05-e759-4931-a47d-ea313d6bbb87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:44:48.463195 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.463179    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b02cd05-e759-4931-a47d-ea313d6bbb87-kube-api-access-5c66k" (OuterVolumeSpecName: "kube-api-access-5c66k") pod "4b02cd05-e759-4931-a47d-ea313d6bbb87" (UID: "4b02cd05-e759-4931-a47d-ea313d6bbb87"). InnerVolumeSpecName "kube-api-access-5c66k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:44:48.463265 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.463246    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8cfdd9-5bc3-4ef5-8993-4999d232a24f-kube-api-access-mnb8t" (OuterVolumeSpecName: "kube-api-access-mnb8t") pod "6c8cfdd9-5bc3-4ef5-8993-4999d232a24f" (UID: "6c8cfdd9-5bc3-4ef5-8993-4999d232a24f"). InnerVolumeSpecName "kube-api-access-mnb8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:44:48.562276 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.562214    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b02cd05-e759-4931-a47d-ea313d6bbb87-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:44:48.562276 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.562236    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnb8t\" (UniqueName: \"kubernetes.io/projected/6c8cfdd9-5bc3-4ef5-8993-4999d232a24f-kube-api-access-mnb8t\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:44:48.562276 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.562245    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c66k\" (UniqueName: \"kubernetes.io/projected/4b02cd05-e759-4931-a47d-ea313d6bbb87-kube-api-access-5c66k\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:44:48.562276 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.562255    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8cfdd9-5bc3-4ef5-8993-4999d232a24f-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:44:48.945235 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.945201    2557 generic.go:356] "Generic (PLEG): container finished" podID="6c8cfdd9-5bc3-4ef5-8993-4999d232a24f" containerID="b98739d40ed453d11d8e3349c1a5eeafd25ebca5cbf6784d5e4b36c0ff13037b" exitCode=0
Sep 08 16:44:48.945657 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.945257    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq"
Sep 08 16:44:48.945657 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.945282    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq" event={"ID":"6c8cfdd9-5bc3-4ef5-8993-4999d232a24f","Type":"ContainerDied","Data":"b98739d40ed453d11d8e3349c1a5eeafd25ebca5cbf6784d5e4b36c0ff13037b"}
Sep 08 16:44:48.945657 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.945370    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq" event={"ID":"6c8cfdd9-5bc3-4ef5-8993-4999d232a24f","Type":"ContainerDied","Data":"5c37993e11d1742d03dd1e0b73cca9fda17458aa88e7c208749c5ab8b25110c7"}
Sep 08 16:44:48.945657 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.945397    2557 scope.go:117] "RemoveContainer" containerID="b98739d40ed453d11d8e3349c1a5eeafd25ebca5cbf6784d5e4b36c0ff13037b"
Sep 08 16:44:48.946749 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.946692    2557 generic.go:356] "Generic (PLEG): container finished" podID="4b02cd05-e759-4931-a47d-ea313d6bbb87" containerID="77a594bac5526ac325200de35641f2753d6f1c23db22619b83cda2e1a1ca871f" exitCode=0
Sep 08 16:44:48.946852 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.946764    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq" event={"ID":"4b02cd05-e759-4931-a47d-ea313d6bbb87","Type":"ContainerDied","Data":"77a594bac5526ac325200de35641f2753d6f1c23db22619b83cda2e1a1ca871f"}
Sep 08 16:44:48.946852 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.946792    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq"
Sep 08 16:44:48.946852 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.946816    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq" event={"ID":"4b02cd05-e759-4931-a47d-ea313d6bbb87","Type":"ContainerDied","Data":"0b2743a5a5788e91192a75802faa7715dbc32176f2dac2d908c1990fb1e9d9e4"}
Sep 08 16:44:48.954931 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.954722    2557 scope.go:117] "RemoveContainer" containerID="b98739d40ed453d11d8e3349c1a5eeafd25ebca5cbf6784d5e4b36c0ff13037b"
Sep 08 16:44:48.955248 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:44:48.955045    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b98739d40ed453d11d8e3349c1a5eeafd25ebca5cbf6784d5e4b36c0ff13037b\": container with ID starting with b98739d40ed453d11d8e3349c1a5eeafd25ebca5cbf6784d5e4b36c0ff13037b not found: ID does not exist" containerID="b98739d40ed453d11d8e3349c1a5eeafd25ebca5cbf6784d5e4b36c0ff13037b"
Sep 08 16:44:48.955248 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.955076    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98739d40ed453d11d8e3349c1a5eeafd25ebca5cbf6784d5e4b36c0ff13037b"} err="failed to get container status \"b98739d40ed453d11d8e3349c1a5eeafd25ebca5cbf6784d5e4b36c0ff13037b\": rpc error: code = NotFound desc = could not find container \"b98739d40ed453d11d8e3349c1a5eeafd25ebca5cbf6784d5e4b36c0ff13037b\": container with ID starting with b98739d40ed453d11d8e3349c1a5eeafd25ebca5cbf6784d5e4b36c0ff13037b not found: ID does not exist"
Sep 08 16:44:48.955248 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.955098    2557 scope.go:117] "RemoveContainer" containerID="77a594bac5526ac325200de35641f2753d6f1c23db22619b83cda2e1a1ca871f"
Sep 08 16:44:48.963247 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.963229    2557 scope.go:117] "RemoveContainer" containerID="77a594bac5526ac325200de35641f2753d6f1c23db22619b83cda2e1a1ca871f"
Sep 08 16:44:48.963537 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:44:48.963514    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77a594bac5526ac325200de35641f2753d6f1c23db22619b83cda2e1a1ca871f\": container with ID starting with 77a594bac5526ac325200de35641f2753d6f1c23db22619b83cda2e1a1ca871f not found: ID does not exist" containerID="77a594bac5526ac325200de35641f2753d6f1c23db22619b83cda2e1a1ca871f"
Sep 08 16:44:48.963621 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.963541    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77a594bac5526ac325200de35641f2753d6f1c23db22619b83cda2e1a1ca871f"} err="failed to get container status \"77a594bac5526ac325200de35641f2753d6f1c23db22619b83cda2e1a1ca871f\": rpc error: code = NotFound desc = could not find container \"77a594bac5526ac325200de35641f2753d6f1c23db22619b83cda2e1a1ca871f\": container with ID starting with 77a594bac5526ac325200de35641f2753d6f1c23db22619b83cda2e1a1ca871f not found: ID does not exist"
Sep 08 16:44:48.969365 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.969343    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq"]
Sep 08 16:44:48.974031 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.973780    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fc1daf8b-744d864cd6-cqkmq"]
Sep 08 16:44:48.985692 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.985668    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq"]
Sep 08 16:44:48.988118 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:48.988098    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fc1daf8b-bd7b9cbd6-klqlq"]
Sep 08 16:44:50.347132 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:50.347099    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b02cd05-e759-4931-a47d-ea313d6bbb87" path="/var/lib/kubelet/pods/4b02cd05-e759-4931-a47d-ea313d6bbb87/volumes"
Sep 08 16:44:50.347627 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:50.347605    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c8cfdd9-5bc3-4ef5-8993-4999d232a24f" path="/var/lib/kubelet/pods/6c8cfdd9-5bc3-4ef5-8993-4999d232a24f/volumes"
Sep 08 16:44:53.210593 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:53.210559    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l"]
Sep 08 16:44:53.210963 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:53.210949    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c8cfdd9-5bc3-4ef5-8993-4999d232a24f" containerName="payload-pre-processing"
Sep 08 16:44:53.210963 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:53.210961    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c8cfdd9-5bc3-4ef5-8993-4999d232a24f" containerName="payload-pre-processing"
Sep 08 16:44:53.211030 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:53.210979    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b02cd05-e759-4931-a47d-ea313d6bbb87" containerName="payload-processing"
Sep 08 16:44:53.211030 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:53.210986    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b02cd05-e759-4931-a47d-ea313d6bbb87" containerName="payload-processing"
Sep 08 16:44:53.211095 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:53.211051    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b02cd05-e759-4931-a47d-ea313d6bbb87" containerName="payload-processing"
Sep 08 16:44:53.211095 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:53.211062    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c8cfdd9-5bc3-4ef5-8993-4999d232a24f" containerName="payload-pre-processing"
Sep 08 16:44:53.214040 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:53.214022    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l"
Sep 08 16:44:53.217153 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:53.217129    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-30506528-dockercfg-tl8tr\""
Sep 08 16:44:53.217153 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:53.217146    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-30506528\""
Sep 08 16:44:53.224199 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:53.224180    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l"]
Sep 08 16:44:53.304512 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:53.304486    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjc96\" (UniqueName: \"kubernetes.io/projected/f7bb16b6-f188-4a0c-8f93-398f9b6ecf29-kube-api-access-vjc96\") pod \"payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l\" (UID: \"f7bb16b6-f188-4a0c-8f93-398f9b6ecf29\") " pod="openshift-ingress/payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l"
Sep 08 16:44:53.304643 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:53.304520    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bb16b6-f188-4a0c-8f93-398f9b6ecf29-plugins-config-volume\") pod \"payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l\" (UID: \"f7bb16b6-f188-4a0c-8f93-398f9b6ecf29\") " pod="openshift-ingress/payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l"
Sep 08 16:44:53.405502 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:53.405471    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjc96\" (UniqueName: \"kubernetes.io/projected/f7bb16b6-f188-4a0c-8f93-398f9b6ecf29-kube-api-access-vjc96\") pod \"payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l\" (UID: \"f7bb16b6-f188-4a0c-8f93-398f9b6ecf29\") " pod="openshift-ingress/payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l"
Sep 08 16:44:53.405743 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:53.405510    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bb16b6-f188-4a0c-8f93-398f9b6ecf29-plugins-config-volume\") pod \"payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l\" (UID: \"f7bb16b6-f188-4a0c-8f93-398f9b6ecf29\") " pod="openshift-ingress/payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l"
Sep 08 16:44:53.406249 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:53.406229    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bb16b6-f188-4a0c-8f93-398f9b6ecf29-plugins-config-volume\") pod \"payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l\" (UID: \"f7bb16b6-f188-4a0c-8f93-398f9b6ecf29\") " pod="openshift-ingress/payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l"
Sep 08 16:44:53.414760 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:53.414740    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjc96\" (UniqueName: \"kubernetes.io/projected/f7bb16b6-f188-4a0c-8f93-398f9b6ecf29-kube-api-access-vjc96\") pod \"payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l\" (UID: \"f7bb16b6-f188-4a0c-8f93-398f9b6ecf29\") " pod="openshift-ingress/payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l"
Sep 08 16:44:53.524635 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:53.524577    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l"
Sep 08 16:44:53.800986 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:53.800961    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l"]
Sep 08 16:44:53.802972 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:44:53.802947    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bb16b6_f188_4a0c_8f93_398f9b6ecf29.slice/crio-41311b7c47042344dd815bdf52546e95f80fff513f7a76b91c917aee45abc00f WatchSource:0}: Error finding container 41311b7c47042344dd815bdf52546e95f80fff513f7a76b91c917aee45abc00f: Status 404 returned error can't find the container with id 41311b7c47042344dd815bdf52546e95f80fff513f7a76b91c917aee45abc00f
Sep 08 16:44:53.976888 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:53.976822    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l" event={"ID":"f7bb16b6-f188-4a0c-8f93-398f9b6ecf29","Type":"ContainerStarted","Data":"f57a1212de410228049ef088e10c429ffa205a44f1548f0db48b85fd316d1e8e"}
Sep 08 16:44:53.977059 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:53.976904    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l" event={"ID":"f7bb16b6-f188-4a0c-8f93-398f9b6ecf29","Type":"ContainerStarted","Data":"41311b7c47042344dd815bdf52546e95f80fff513f7a76b91c917aee45abc00f"}
Sep 08 16:44:53.977059 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:53.976956    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l"
Sep 08 16:44:53.995450 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:53.995405    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l" podStartSLOduration=0.995368971 podStartE2EDuration="995.368971ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:44:53.993973462 +0000 UTC m=+878.156944909" watchObservedRunningTime="2026-09-08 16:44:53.995368971 +0000 UTC m=+878.158340419"
Sep 08 16:44:56.949182 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:56.949143    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:44:56.949654 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:56.949202    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:44:56.961609 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:56.961579    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:44:57.001782 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:57.001750    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:44:57.744502 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:44:57.744468    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:45:00.138671 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:00.138630    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4cea93-29814765-kx9lw"]
Sep 08 16:45:00.141181 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:00.141164    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4cea93-29814765-kx9lw"
Sep 08 16:45:00.143749 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:00.143724    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-886jf\""
Sep 08 16:45:00.145786 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:00.145762    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-43a905-29814765-b6jrv"]
Sep 08 16:45:00.148863 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:00.148821    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814765-gmpx8"]
Sep 08 16:45:00.149066 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:00.148934    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-43a905-29814765-b6jrv"
Sep 08 16:45:00.151096 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:00.151079    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814765-gmpx8"
Sep 08 16:45:00.151279 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:00.151245    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4cea93-29814765-kx9lw"]
Sep 08 16:45:00.158581 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:00.158556    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814765-gmpx8"]
Sep 08 16:45:00.160800 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:00.160772    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-43a905-29814765-b6jrv"]
Sep 08 16:45:00.180184 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:00.180158    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4cea93-29814765-kx9lw"
Sep 08 16:45:00.189324 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:00.189299    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-43a905-29814765-b6jrv"
Sep 08 16:45:00.220335 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:00.220314    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814765-gmpx8"
Sep 08 16:45:00.455646 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:00.455598    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4cea93-29814765-kx9lw"]
Sep 08 16:45:00.458531 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:45:00.458499    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db57c86_99cc_4cbb_9ad3_d2f897b9afcc.slice/crio-0376d4be535d00ae0b7d3d25b987b7b1943afef9a92b840e3af408ccf653454c WatchSource:0}: Error finding container 0376d4be535d00ae0b7d3d25b987b7b1943afef9a92b840e3af408ccf653454c: Status 404 returned error can't find the container with id 0376d4be535d00ae0b7d3d25b987b7b1943afef9a92b840e3af408ccf653454c
Sep 08 16:45:00.468282 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:00.468248    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-43a905-29814765-b6jrv"]
Sep 08 16:45:00.470173 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:45:00.470146    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc153a0c2_da40_41a1_944c_9a7e3df669b8.slice/crio-d23b60cec1ec4a17faedc150b1b9d741afd45a47c6ba007a57c0b305c817aeae WatchSource:0}: Error finding container d23b60cec1ec4a17faedc150b1b9d741afd45a47c6ba007a57c0b305c817aeae: Status 404 returned error can't find the container with id d23b60cec1ec4a17faedc150b1b9d741afd45a47c6ba007a57c0b305c817aeae
Sep 08 16:45:00.518560 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:00.518538    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814765-gmpx8"]
Sep 08 16:45:00.520995 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:45:00.520970    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5fdae75_6aee_47b9_9a3c_21e2368ccf32.slice/crio-f74725bed237efc334688810d0580cdc5da54773d65ec58a2df6222b6ea6e4c7 WatchSource:0}: Error finding container f74725bed237efc334688810d0580cdc5da54773d65ec58a2df6222b6ea6e4c7: Status 404 returned error can't find the container with id f74725bed237efc334688810d0580cdc5da54773d65ec58a2df6222b6ea6e4c7
Sep 08 16:45:01.012656 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:01.012610    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4cea93-29814765-kx9lw" event={"ID":"1db57c86-99cc-4cbb-9ad3-d2f897b9afcc","Type":"ContainerStarted","Data":"00ddc0352ff58783054843f57645ba4db1bb577ec726bd2600cf41bae7462796"}
Sep 08 16:45:01.012911 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:01.012678    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4cea93-29814765-kx9lw" event={"ID":"1db57c86-99cc-4cbb-9ad3-d2f897b9afcc","Type":"ContainerStarted","Data":"0376d4be535d00ae0b7d3d25b987b7b1943afef9a92b840e3af408ccf653454c"}
Sep 08 16:45:01.014240 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:01.014207    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814765-gmpx8" event={"ID":"f5fdae75-6aee-47b9-9a3c-21e2368ccf32","Type":"ContainerStarted","Data":"49cbdce4240960b6c7de6da8822ce915c9e9772fa12a42eaa324c3d7703310f4"}
Sep 08 16:45:01.014368 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:01.014250    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814765-gmpx8" event={"ID":"f5fdae75-6aee-47b9-9a3c-21e2368ccf32","Type":"ContainerStarted","Data":"f74725bed237efc334688810d0580cdc5da54773d65ec58a2df6222b6ea6e4c7"}
Sep 08 16:45:01.015709 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:01.015689    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-43a905-29814765-b6jrv" event={"ID":"c153a0c2-da40-41a1-944c-9a7e3df669b8","Type":"ContainerStarted","Data":"4fff6c1cea2bb2c759129e9987721ad38f975bdea7824b6f206aedf24439a164"}
Sep 08 16:45:01.015800 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:01.015725    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-43a905-29814765-b6jrv" event={"ID":"c153a0c2-da40-41a1-944c-9a7e3df669b8","Type":"ContainerStarted","Data":"d23b60cec1ec4a17faedc150b1b9d741afd45a47c6ba007a57c0b305c817aeae"}
Sep 08 16:45:01.027378 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:01.027304    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4cea93-29814765-kx9lw" podStartSLOduration=1.027291094 podStartE2EDuration="1.027291094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:45:01.026771853 +0000 UTC m=+885.189743300" watchObservedRunningTime="2026-09-08 16:45:01.027291094 +0000 UTC m=+885.190262541"
Sep 08 16:45:01.044882 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:01.044825    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814765-gmpx8" podStartSLOduration=1.044814673 podStartE2EDuration="1.044814673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:45:01.043401439 +0000 UTC m=+885.206372887" watchObservedRunningTime="2026-09-08 16:45:01.044814673 +0000 UTC m=+885.207786120"
Sep 08 16:45:01.057648 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:01.057606    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-43a905-29814765-b6jrv" podStartSLOduration=1.057593439 podStartE2EDuration="1.057593439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:45:01.056934536 +0000 UTC m=+885.219905986" watchObservedRunningTime="2026-09-08 16:45:01.057593439 +0000 UTC m=+885.220564886"
Sep 08 16:45:02.022671 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:02.022633    2557 generic.go:356] "Generic (PLEG): container finished" podID="1db57c86-99cc-4cbb-9ad3-d2f897b9afcc" containerID="00ddc0352ff58783054843f57645ba4db1bb577ec726bd2600cf41bae7462796" exitCode=0
Sep 08 16:45:02.023111 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:02.022720    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4cea93-29814765-kx9lw" event={"ID":"1db57c86-99cc-4cbb-9ad3-d2f897b9afcc","Type":"ContainerDied","Data":"00ddc0352ff58783054843f57645ba4db1bb577ec726bd2600cf41bae7462796"}
Sep 08 16:45:02.024297 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:02.024273    2557 generic.go:356] "Generic (PLEG): container finished" podID="f5fdae75-6aee-47b9-9a3c-21e2368ccf32" containerID="49cbdce4240960b6c7de6da8822ce915c9e9772fa12a42eaa324c3d7703310f4" exitCode=0
Sep 08 16:45:02.024398 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:02.024354    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814765-gmpx8" event={"ID":"f5fdae75-6aee-47b9-9a3c-21e2368ccf32","Type":"ContainerDied","Data":"49cbdce4240960b6c7de6da8822ce915c9e9772fa12a42eaa324c3d7703310f4"}
Sep 08 16:45:02.025671 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:02.025654    2557 generic.go:356] "Generic (PLEG): container finished" podID="c153a0c2-da40-41a1-944c-9a7e3df669b8" containerID="4fff6c1cea2bb2c759129e9987721ad38f975bdea7824b6f206aedf24439a164" exitCode=0
Sep 08 16:45:02.025767 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:02.025679    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-43a905-29814765-b6jrv" event={"ID":"c153a0c2-da40-41a1-944c-9a7e3df669b8","Type":"ContainerDied","Data":"4fff6c1cea2bb2c759129e9987721ad38f975bdea7824b6f206aedf24439a164"}
Sep 08 16:45:03.174633 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:03.174241    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-43a905-29814765-b6jrv"
Sep 08 16:45:03.204777 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:03.204748    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4cea93-29814765-kx9lw"
Sep 08 16:45:03.247775 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:03.247750    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814765-gmpx8"
Sep 08 16:45:04.036561 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:04.036530    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4cea93-29814765-kx9lw"
Sep 08 16:45:04.036723 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:04.036534    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4cea93-29814765-kx9lw" event={"ID":"1db57c86-99cc-4cbb-9ad3-d2f897b9afcc","Type":"ContainerDied","Data":"0376d4be535d00ae0b7d3d25b987b7b1943afef9a92b840e3af408ccf653454c"}
Sep 08 16:45:04.036723 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:04.036685    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0376d4be535d00ae0b7d3d25b987b7b1943afef9a92b840e3af408ccf653454c"
Sep 08 16:45:04.037975 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:04.037956    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814765-gmpx8"
Sep 08 16:45:04.038100 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:04.037957    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814765-gmpx8" event={"ID":"f5fdae75-6aee-47b9-9a3c-21e2368ccf32","Type":"ContainerDied","Data":"f74725bed237efc334688810d0580cdc5da54773d65ec58a2df6222b6ea6e4c7"}
Sep 08 16:45:04.038100 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:04.038073    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f74725bed237efc334688810d0580cdc5da54773d65ec58a2df6222b6ea6e4c7"
Sep 08 16:45:04.039295 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:04.039277    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-43a905-29814765-b6jrv" event={"ID":"c153a0c2-da40-41a1-944c-9a7e3df669b8","Type":"ContainerDied","Data":"d23b60cec1ec4a17faedc150b1b9d741afd45a47c6ba007a57c0b305c817aeae"}
Sep 08 16:45:04.039295 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:04.039287    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-43a905-29814765-b6jrv"
Sep 08 16:45:04.039425 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:04.039313    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d23b60cec1ec4a17faedc150b1b9d741afd45a47c6ba007a57c0b305c817aeae"
Sep 08 16:45:04.983999 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:04.983966    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l"
Sep 08 16:45:17.147604 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:17.147574    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:45:17.270481 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:17.270444    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30506528-gnnbj"]
Sep 08 16:45:17.271032 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:17.271011    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1db57c86-99cc-4cbb-9ad3-d2f897b9afcc" containerName="cleanup"
Sep 08 16:45:17.271032 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:17.271033    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1db57c86-99cc-4cbb-9ad3-d2f897b9afcc" containerName="cleanup"
Sep 08 16:45:17.271209 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:17.271044    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c153a0c2-da40-41a1-944c-9a7e3df669b8" containerName="cleanup"
Sep 08 16:45:17.271209 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:17.271051    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c153a0c2-da40-41a1-944c-9a7e3df669b8" containerName="cleanup"
Sep 08 16:45:17.271209 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:17.271085    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5fdae75-6aee-47b9-9a3c-21e2368ccf32" containerName="cleanup"
Sep 08 16:45:17.271209 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:17.271094    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5fdae75-6aee-47b9-9a3c-21e2368ccf32" containerName="cleanup"
Sep 08 16:45:17.271209 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:17.271182    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1db57c86-99cc-4cbb-9ad3-d2f897b9afcc" containerName="cleanup"
Sep 08 16:45:17.271209 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:17.271194    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c153a0c2-da40-41a1-944c-9a7e3df669b8" containerName="cleanup"
Sep 08 16:45:17.271209 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:17.271203    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5fdae75-6aee-47b9-9a3c-21e2368ccf32" containerName="cleanup"
Sep 08 16:45:17.274661 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:17.274637    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30506528-gnnbj"
Sep 08 16:45:17.277899 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:17.277871    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:45:17.278012 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:17.277913    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-886jf\""
Sep 08 16:45:17.282762 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:17.282669    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30506528-gnnbj"]
Sep 08 16:45:17.327432 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:17.327395    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa3f6b0-f4e3-49d6-9164-1151f2ced41a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-30506528-gnnbj\" (UID: \"1aa3f6b0-f4e3-49d6-9164-1151f2ced41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30506528-gnnbj"
Sep 08 16:45:17.427093 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:17.427060    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"]
Sep 08 16:45:17.427819 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:17.427795    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa3f6b0-f4e3-49d6-9164-1151f2ced41a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-30506528-gnnbj\" (UID: \"1aa3f6b0-f4e3-49d6-9164-1151f2ced41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30506528-gnnbj"
Sep 08 16:45:17.428473 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:17.428444    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa3f6b0-f4e3-49d6-9164-1151f2ced41a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-30506528-gnnbj\" (UID: \"1aa3f6b0-f4e3-49d6-9164-1151f2ced41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30506528-gnnbj"
Sep 08 16:45:17.588063 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:17.588030    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30506528-gnnbj"
Sep 08 16:45:17.856942 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:17.856918    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30506528-gnnbj"]
Sep 08 16:45:17.858748 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:45:17.858717    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa3f6b0_f4e3_49d6_9164_1151f2ced41a.slice/crio-e9eb719c5903768b9799ebf0f8068182788ea59ed6a1de7cbd2d0094af8b0192 WatchSource:0}: Error finding container e9eb719c5903768b9799ebf0f8068182788ea59ed6a1de7cbd2d0094af8b0192: Status 404 returned error can't find the container with id e9eb719c5903768b9799ebf0f8068182788ea59ed6a1de7cbd2d0094af8b0192
Sep 08 16:45:18.112675 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:18.112609    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30506528-gnnbj" event={"ID":"1aa3f6b0-f4e3-49d6-9164-1151f2ced41a","Type":"ContainerStarted","Data":"e10c79ce0382bdfce579c4c18ac5661870c670ec8497b031faf8a485fe6b70c4"}
Sep 08 16:45:18.112675 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:18.112672    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30506528-gnnbj" event={"ID":"1aa3f6b0-f4e3-49d6-9164-1151f2ced41a","Type":"ContainerStarted","Data":"e9eb719c5903768b9799ebf0f8068182788ea59ed6a1de7cbd2d0094af8b0192"}
Sep 08 16:45:18.132715 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:18.132667    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30506528-gnnbj" podStartSLOduration=1.132652813 podStartE2EDuration="1.132652813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:45:18.131501724 +0000 UTC m=+902.294473184" watchObservedRunningTime="2026-09-08 16:45:18.132652813 +0000 UTC m=+902.295624308"
Sep 08 16:45:18.295647 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:18.295615    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4cea93-gmlw4"]
Sep 08 16:45:18.297955 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:18.297931    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4cea93-gmlw4"
Sep 08 16:45:18.308232 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:18.308204    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4cea93-gmlw4"]
Sep 08 16:45:18.335227 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:18.335204    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64a18e8-3dc0-48c3-8a09-189894e4b9ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-4cea93-gmlw4\" (UID: \"a64a18e8-3dc0-48c3-8a09-189894e4b9ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4cea93-gmlw4"
Sep 08 16:45:18.436345 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:18.436318    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64a18e8-3dc0-48c3-8a09-189894e4b9ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-4cea93-gmlw4\" (UID: \"a64a18e8-3dc0-48c3-8a09-189894e4b9ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4cea93-gmlw4"
Sep 08 16:45:18.436914 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:18.436896    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64a18e8-3dc0-48c3-8a09-189894e4b9ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-4cea93-gmlw4\" (UID: \"a64a18e8-3dc0-48c3-8a09-189894e4b9ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4cea93-gmlw4"
Sep 08 16:45:18.609477 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:18.609451    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4cea93-gmlw4"
Sep 08 16:45:18.886222 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:18.886191    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4cea93-gmlw4"]
Sep 08 16:45:18.887957 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:45:18.887916    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda64a18e8_3dc0_48c3_8a09_189894e4b9ab.slice/crio-3c8118b7de42f9e46ddec2b3faa4d34804c076fb5e0a2f997d5dc2225f9e95d2 WatchSource:0}: Error finding container 3c8118b7de42f9e46ddec2b3faa4d34804c076fb5e0a2f997d5dc2225f9e95d2: Status 404 returned error can't find the container with id 3c8118b7de42f9e46ddec2b3faa4d34804c076fb5e0a2f997d5dc2225f9e95d2
Sep 08 16:45:18.922731 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:18.922705    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"]
Sep 08 16:45:19.119232 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:19.119203    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4cea93-gmlw4" event={"ID":"a64a18e8-3dc0-48c3-8a09-189894e4b9ab","Type":"ContainerStarted","Data":"3c8118b7de42f9e46ddec2b3faa4d34804c076fb5e0a2f997d5dc2225f9e95d2"}
Sep 08 16:45:19.120733 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:19.120709    2557 generic.go:356] "Generic (PLEG): container finished" podID="1aa3f6b0-f4e3-49d6-9164-1151f2ced41a" containerID="e10c79ce0382bdfce579c4c18ac5661870c670ec8497b031faf8a485fe6b70c4" exitCode=0
Sep 08 16:45:19.120866 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:19.120756    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30506528-gnnbj" event={"ID":"1aa3f6b0-f4e3-49d6-9164-1151f2ced41a","Type":"ContainerDied","Data":"e10c79ce0382bdfce579c4c18ac5661870c670ec8497b031faf8a485fe6b70c4"}
Sep 08 16:45:19.717583 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:19.717554    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"]
Sep 08 16:45:20.127760 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:20.127671    2557 generic.go:356] "Generic (PLEG): container finished" podID="a64a18e8-3dc0-48c3-8a09-189894e4b9ab" containerID="83732e6fd1a9cfffaa3c527e8c6e58f19f1d49c6bd171d8fcf4f2796f36d27c1" exitCode=0
Sep 08 16:45:20.127901 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:20.127762    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4cea93-gmlw4" event={"ID":"a64a18e8-3dc0-48c3-8a09-189894e4b9ab","Type":"ContainerDied","Data":"83732e6fd1a9cfffaa3c527e8c6e58f19f1d49c6bd171d8fcf4f2796f36d27c1"}
Sep 08 16:45:20.265024 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:20.265003    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30506528-gnnbj"
Sep 08 16:45:20.354888 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:20.354855    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa3f6b0-f4e3-49d6-9164-1151f2ced41a-maas-api-service-ca\") pod \"1aa3f6b0-f4e3-49d6-9164-1151f2ced41a\" (UID: \"1aa3f6b0-f4e3-49d6-9164-1151f2ced41a\") "
Sep 08 16:45:20.355097 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:20.355077    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa3f6b0-f4e3-49d6-9164-1151f2ced41a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1aa3f6b0-f4e3-49d6-9164-1151f2ced41a" (UID: "1aa3f6b0-f4e3-49d6-9164-1151f2ced41a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:45:20.455808 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:20.455777    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa3f6b0-f4e3-49d6-9164-1151f2ced41a-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:21.133859 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:21.133811    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30506528-gnnbj"
Sep 08 16:45:21.133859 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:21.133819    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30506528-gnnbj" event={"ID":"1aa3f6b0-f4e3-49d6-9164-1151f2ced41a","Type":"ContainerDied","Data":"e9eb719c5903768b9799ebf0f8068182788ea59ed6a1de7cbd2d0094af8b0192"}
Sep 08 16:45:21.134346 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:21.133900    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9eb719c5903768b9799ebf0f8068182788ea59ed6a1de7cbd2d0094af8b0192"
Sep 08 16:45:21.289625 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:21.289596    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4cea93-gmlw4"
Sep 08 16:45:21.364265 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:21.364236    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64a18e8-3dc0-48c3-8a09-189894e4b9ab-maas-api-service-ca\") pod \"a64a18e8-3dc0-48c3-8a09-189894e4b9ab\" (UID: \"a64a18e8-3dc0-48c3-8a09-189894e4b9ab\") "
Sep 08 16:45:21.364610 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:21.364584    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a64a18e8-3dc0-48c3-8a09-189894e4b9ab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a64a18e8-3dc0-48c3-8a09-189894e4b9ab" (UID: "a64a18e8-3dc0-48c3-8a09-189894e4b9ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:45:21.465587 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:21.465557    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a64a18e8-3dc0-48c3-8a09-189894e4b9ab-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:22.140717 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:22.140683    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4cea93-gmlw4" event={"ID":"a64a18e8-3dc0-48c3-8a09-189894e4b9ab","Type":"ContainerDied","Data":"3c8118b7de42f9e46ddec2b3faa4d34804c076fb5e0a2f997d5dc2225f9e95d2"}
Sep 08 16:45:22.141156 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:22.140735    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c8118b7de42f9e46ddec2b3faa4d34804c076fb5e0a2f997d5dc2225f9e95d2"
Sep 08 16:45:22.141156 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:22.140699    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4cea93-gmlw4"
Sep 08 16:45:23.649230 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:23.649197    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l"]
Sep 08 16:45:23.649651 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:23.649457    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l" podUID="f7bb16b6-f188-4a0c-8f93-398f9b6ecf29" containerName="payload-processing" containerID="cri-o://f57a1212de410228049ef088e10c429ffa205a44f1548f0db48b85fd316d1e8e" gracePeriod=30
Sep 08 16:45:23.869823 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:23.869790    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30506528-gnnbj"]
Sep 08 16:45:23.872288 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:23.872259    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30506528-gnnbj"]
Sep 08 16:45:23.907409 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:23.907356    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l"
Sep 08 16:45:23.989193 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:23.989159    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjc96\" (UniqueName: \"kubernetes.io/projected/f7bb16b6-f188-4a0c-8f93-398f9b6ecf29-kube-api-access-vjc96\") pod \"f7bb16b6-f188-4a0c-8f93-398f9b6ecf29\" (UID: \"f7bb16b6-f188-4a0c-8f93-398f9b6ecf29\") "
Sep 08 16:45:23.989323 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:23.989230    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bb16b6-f188-4a0c-8f93-398f9b6ecf29-plugins-config-volume\") pod \"f7bb16b6-f188-4a0c-8f93-398f9b6ecf29\" (UID: \"f7bb16b6-f188-4a0c-8f93-398f9b6ecf29\") "
Sep 08 16:45:23.989617 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:23.989592    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7bb16b6-f188-4a0c-8f93-398f9b6ecf29-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f7bb16b6-f188-4a0c-8f93-398f9b6ecf29" (UID: "f7bb16b6-f188-4a0c-8f93-398f9b6ecf29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:45:23.991423 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:23.991403    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7bb16b6-f188-4a0c-8f93-398f9b6ecf29-kube-api-access-vjc96" (OuterVolumeSpecName: "kube-api-access-vjc96") pod "f7bb16b6-f188-4a0c-8f93-398f9b6ecf29" (UID: "f7bb16b6-f188-4a0c-8f93-398f9b6ecf29"). InnerVolumeSpecName "kube-api-access-vjc96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:45:24.090136 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:24.090102    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjc96\" (UniqueName: \"kubernetes.io/projected/f7bb16b6-f188-4a0c-8f93-398f9b6ecf29-kube-api-access-vjc96\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:24.090136 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:24.090131    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bb16b6-f188-4a0c-8f93-398f9b6ecf29-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:24.156300 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:24.156270    2557 generic.go:356] "Generic (PLEG): container finished" podID="f7bb16b6-f188-4a0c-8f93-398f9b6ecf29" containerID="f57a1212de410228049ef088e10c429ffa205a44f1548f0db48b85fd316d1e8e" exitCode=0
Sep 08 16:45:24.156567 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:24.156317    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l"
Sep 08 16:45:24.156567 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:24.156359    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l" event={"ID":"f7bb16b6-f188-4a0c-8f93-398f9b6ecf29","Type":"ContainerDied","Data":"f57a1212de410228049ef088e10c429ffa205a44f1548f0db48b85fd316d1e8e"}
Sep 08 16:45:24.156567 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:24.156415    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l" event={"ID":"f7bb16b6-f188-4a0c-8f93-398f9b6ecf29","Type":"ContainerDied","Data":"41311b7c47042344dd815bdf52546e95f80fff513f7a76b91c917aee45abc00f"}
Sep 08 16:45:24.156567 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:24.156439    2557 scope.go:117] "RemoveContainer" containerID="f57a1212de410228049ef088e10c429ffa205a44f1548f0db48b85fd316d1e8e"
Sep 08 16:45:24.165276 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:24.165260    2557 scope.go:117] "RemoveContainer" containerID="f57a1212de410228049ef088e10c429ffa205a44f1548f0db48b85fd316d1e8e"
Sep 08 16:45:24.165500 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:45:24.165485    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f57a1212de410228049ef088e10c429ffa205a44f1548f0db48b85fd316d1e8e\": container with ID starting with f57a1212de410228049ef088e10c429ffa205a44f1548f0db48b85fd316d1e8e not found: ID does not exist" containerID="f57a1212de410228049ef088e10c429ffa205a44f1548f0db48b85fd316d1e8e"
Sep 08 16:45:24.165562 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:24.165508    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f57a1212de410228049ef088e10c429ffa205a44f1548f0db48b85fd316d1e8e"} err="failed to get container status \"f57a1212de410228049ef088e10c429ffa205a44f1548f0db48b85fd316d1e8e\": rpc error: code = NotFound desc = could not find container \"f57a1212de410228049ef088e10c429ffa205a44f1548f0db48b85fd316d1e8e\": container with ID starting with f57a1212de410228049ef088e10c429ffa205a44f1548f0db48b85fd316d1e8e not found: ID does not exist"
Sep 08 16:45:24.177315 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:24.177286    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l"]
Sep 08 16:45:24.181088 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:24.181069    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-30506528-6dd6468d45-vkv2l"]
Sep 08 16:45:24.346080 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:24.346049    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1aa3f6b0-f4e3-49d6-9164-1151f2ced41a" path="/var/lib/kubelet/pods/1aa3f6b0-f4e3-49d6-9164-1151f2ced41a/volumes"
Sep 08 16:45:24.346396 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:24.346384    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7bb16b6-f188-4a0c-8f93-398f9b6ecf29" path="/var/lib/kubelet/pods/f7bb16b6-f188-4a0c-8f93-398f9b6ecf29/volumes"
Sep 08 16:45:29.490969 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:29.490938    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4cea93-29814765-kx9lw"]
Sep 08 16:45:29.493135 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:29.493114    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-4cea93-29814765-kx9lw"]
Sep 08 16:45:29.530977 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:29.530954    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx"]
Sep 08 16:45:29.531175 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:29.531149    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx" podUID="25e24a89-d3b7-4413-95a0-035f73c78cb5" containerName="payload-processing" containerID="cri-o://2d706540e10b6d8576a822f269040a15931c9cb1775a7b760adb96184d3ca9ba" gracePeriod=30
Sep 08 16:45:29.549803 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:29.549778    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq"]
Sep 08 16:45:29.550029 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:29.549992    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq" podUID="67b7f626-d8af-4970-976b-f30ff734dba1" containerName="payload-pre-processing" containerID="cri-o://d572f0ddfdfaa70bee05bd86361659652b86da5d79db7fcc2c9eafb66779eeb6" gracePeriod=30
Sep 08 16:45:29.832070 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:29.832039    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx"
Sep 08 16:45:29.834646 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:29.834625    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq"
Sep 08 16:45:29.939572 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:29.939536    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25e24a89-d3b7-4413-95a0-035f73c78cb5-plugins-config-volume\") pod \"25e24a89-d3b7-4413-95a0-035f73c78cb5\" (UID: \"25e24a89-d3b7-4413-95a0-035f73c78cb5\") "
Sep 08 16:45:29.939741 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:29.939577    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b7f626-d8af-4970-976b-f30ff734dba1-plugins-config-volume\") pod \"67b7f626-d8af-4970-976b-f30ff734dba1\" (UID: \"67b7f626-d8af-4970-976b-f30ff734dba1\") "
Sep 08 16:45:29.939741 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:29.939677    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvjzb\" (UniqueName: \"kubernetes.io/projected/67b7f626-d8af-4970-976b-f30ff734dba1-kube-api-access-mvjzb\") pod \"67b7f626-d8af-4970-976b-f30ff734dba1\" (UID: \"67b7f626-d8af-4970-976b-f30ff734dba1\") "
Sep 08 16:45:29.939741 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:29.939713    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgwmh\" (UniqueName: \"kubernetes.io/projected/25e24a89-d3b7-4413-95a0-035f73c78cb5-kube-api-access-mgwmh\") pod \"25e24a89-d3b7-4413-95a0-035f73c78cb5\" (UID: \"25e24a89-d3b7-4413-95a0-035f73c78cb5\") "
Sep 08 16:45:29.940022 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:29.939991    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e24a89-d3b7-4413-95a0-035f73c78cb5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "25e24a89-d3b7-4413-95a0-035f73c78cb5" (UID: "25e24a89-d3b7-4413-95a0-035f73c78cb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:45:29.940069 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:29.940016    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b7f626-d8af-4970-976b-f30ff734dba1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67b7f626-d8af-4970-976b-f30ff734dba1" (UID: "67b7f626-d8af-4970-976b-f30ff734dba1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:45:29.941845 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:29.941809    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e24a89-d3b7-4413-95a0-035f73c78cb5-kube-api-access-mgwmh" (OuterVolumeSpecName: "kube-api-access-mgwmh") pod "25e24a89-d3b7-4413-95a0-035f73c78cb5" (UID: "25e24a89-d3b7-4413-95a0-035f73c78cb5"). InnerVolumeSpecName "kube-api-access-mgwmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:45:29.941921 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:29.941869    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b7f626-d8af-4970-976b-f30ff734dba1-kube-api-access-mvjzb" (OuterVolumeSpecName: "kube-api-access-mvjzb") pod "67b7f626-d8af-4970-976b-f30ff734dba1" (UID: "67b7f626-d8af-4970-976b-f30ff734dba1"). InnerVolumeSpecName "kube-api-access-mvjzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:45:30.040761 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.040682    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvjzb\" (UniqueName: \"kubernetes.io/projected/67b7f626-d8af-4970-976b-f30ff734dba1-kube-api-access-mvjzb\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:30.040761 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.040709    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgwmh\" (UniqueName: \"kubernetes.io/projected/25e24a89-d3b7-4413-95a0-035f73c78cb5-kube-api-access-mgwmh\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:30.040761 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.040718    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25e24a89-d3b7-4413-95a0-035f73c78cb5-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:30.040761 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.040726    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b7f626-d8af-4970-976b-f30ff734dba1-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:30.193604 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.193566    2557 generic.go:356] "Generic (PLEG): container finished" podID="25e24a89-d3b7-4413-95a0-035f73c78cb5" containerID="2d706540e10b6d8576a822f269040a15931c9cb1775a7b760adb96184d3ca9ba" exitCode=0
Sep 08 16:45:30.193794 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.193625    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx" event={"ID":"25e24a89-d3b7-4413-95a0-035f73c78cb5","Type":"ContainerDied","Data":"2d706540e10b6d8576a822f269040a15931c9cb1775a7b760adb96184d3ca9ba"}
Sep 08 16:45:30.193794 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.193689    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx"
Sep 08 16:45:30.193794 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.193722    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx" event={"ID":"25e24a89-d3b7-4413-95a0-035f73c78cb5","Type":"ContainerDied","Data":"5ced992e5439d2e25dd7fb63b491ab38bbd913c18d6aeeff1cc17bc80ca78367"}
Sep 08 16:45:30.193794 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.193755    2557 scope.go:117] "RemoveContainer" containerID="2d706540e10b6d8576a822f269040a15931c9cb1775a7b760adb96184d3ca9ba"
Sep 08 16:45:30.195246 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.195222    2557 generic.go:356] "Generic (PLEG): container finished" podID="67b7f626-d8af-4970-976b-f30ff734dba1" containerID="d572f0ddfdfaa70bee05bd86361659652b86da5d79db7fcc2c9eafb66779eeb6" exitCode=0
Sep 08 16:45:30.195343 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.195256    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq" event={"ID":"67b7f626-d8af-4970-976b-f30ff734dba1","Type":"ContainerDied","Data":"d572f0ddfdfaa70bee05bd86361659652b86da5d79db7fcc2c9eafb66779eeb6"}
Sep 08 16:45:30.195343 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.195290    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq"
Sep 08 16:45:30.195343 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.195317    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq" event={"ID":"67b7f626-d8af-4970-976b-f30ff734dba1","Type":"ContainerDied","Data":"e9f9c6c43ca9a90d7d9c8e5c983a3981d09ddd9408b6328ed5c9bf12f14ec2ce"}
Sep 08 16:45:30.204828 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.204744    2557 scope.go:117] "RemoveContainer" containerID="2d706540e10b6d8576a822f269040a15931c9cb1775a7b760adb96184d3ca9ba"
Sep 08 16:45:30.205441 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:45:30.205082    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d706540e10b6d8576a822f269040a15931c9cb1775a7b760adb96184d3ca9ba\": container with ID starting with 2d706540e10b6d8576a822f269040a15931c9cb1775a7b760adb96184d3ca9ba not found: ID does not exist" containerID="2d706540e10b6d8576a822f269040a15931c9cb1775a7b760adb96184d3ca9ba"
Sep 08 16:45:30.205441 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.205112    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d706540e10b6d8576a822f269040a15931c9cb1775a7b760adb96184d3ca9ba"} err="failed to get container status \"2d706540e10b6d8576a822f269040a15931c9cb1775a7b760adb96184d3ca9ba\": rpc error: code = NotFound desc = could not find container \"2d706540e10b6d8576a822f269040a15931c9cb1775a7b760adb96184d3ca9ba\": container with ID starting with 2d706540e10b6d8576a822f269040a15931c9cb1775a7b760adb96184d3ca9ba not found: ID does not exist"
Sep 08 16:45:30.205441 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.205135    2557 scope.go:117] "RemoveContainer" containerID="d572f0ddfdfaa70bee05bd86361659652b86da5d79db7fcc2c9eafb66779eeb6"
Sep 08 16:45:30.205982 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.205963    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4cea93-gmlw4"]
Sep 08 16:45:30.208154 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.208134    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-4cea93-gmlw4"]
Sep 08 16:45:30.214861 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.214825    2557 scope.go:117] "RemoveContainer" containerID="d572f0ddfdfaa70bee05bd86361659652b86da5d79db7fcc2c9eafb66779eeb6"
Sep 08 16:45:30.215118 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:45:30.215099    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d572f0ddfdfaa70bee05bd86361659652b86da5d79db7fcc2c9eafb66779eeb6\": container with ID starting with d572f0ddfdfaa70bee05bd86361659652b86da5d79db7fcc2c9eafb66779eeb6 not found: ID does not exist" containerID="d572f0ddfdfaa70bee05bd86361659652b86da5d79db7fcc2c9eafb66779eeb6"
Sep 08 16:45:30.215181 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.215123    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d572f0ddfdfaa70bee05bd86361659652b86da5d79db7fcc2c9eafb66779eeb6"} err="failed to get container status \"d572f0ddfdfaa70bee05bd86361659652b86da5d79db7fcc2c9eafb66779eeb6\": rpc error: code = NotFound desc = could not find container \"d572f0ddfdfaa70bee05bd86361659652b86da5d79db7fcc2c9eafb66779eeb6\": container with ID starting with d572f0ddfdfaa70bee05bd86361659652b86da5d79db7fcc2c9eafb66779eeb6 not found: ID does not exist"
Sep 08 16:45:30.220519 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.220502    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx"]
Sep 08 16:45:30.222149 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.222128    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4cea93-d769fd8c9-c72zx"]
Sep 08 16:45:30.232053 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.232026    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq"]
Sep 08 16:45:30.238032 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.238003    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4cea93-954cc6446-gbskq"]
Sep 08 16:45:30.345653 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.345583    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1db57c86-99cc-4cbb-9ad3-d2f897b9afcc" path="/var/lib/kubelet/pods/1db57c86-99cc-4cbb-9ad3-d2f897b9afcc/volumes"
Sep 08 16:45:30.345917 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.345903    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25e24a89-d3b7-4413-95a0-035f73c78cb5" path="/var/lib/kubelet/pods/25e24a89-d3b7-4413-95a0-035f73c78cb5/volumes"
Sep 08 16:45:30.346207 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.346196    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67b7f626-d8af-4970-976b-f30ff734dba1" path="/var/lib/kubelet/pods/67b7f626-d8af-4970-976b-f30ff734dba1/volumes"
Sep 08 16:45:30.346467 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.346456    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a64a18e8-3dc0-48c3-8a09-189894e4b9ab" path="/var/lib/kubelet/pods/a64a18e8-3dc0-48c3-8a09-189894e4b9ab/volumes"
Sep 08 16:45:30.840189 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.840155    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z"]
Sep 08 16:45:30.840550 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.840489    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a64a18e8-3dc0-48c3-8a09-189894e4b9ab" containerName="revoke-keys"
Sep 08 16:45:30.840550 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.840500    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a64a18e8-3dc0-48c3-8a09-189894e4b9ab" containerName="revoke-keys"
Sep 08 16:45:30.840550 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.840520    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25e24a89-d3b7-4413-95a0-035f73c78cb5" containerName="payload-processing"
Sep 08 16:45:30.840550 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.840525    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="25e24a89-d3b7-4413-95a0-035f73c78cb5" containerName="payload-processing"
Sep 08 16:45:30.840550 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.840539    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b7f626-d8af-4970-976b-f30ff734dba1" containerName="payload-pre-processing"
Sep 08 16:45:30.840550 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.840545    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b7f626-d8af-4970-976b-f30ff734dba1" containerName="payload-pre-processing"
Sep 08 16:45:30.840550 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.840553    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7bb16b6-f188-4a0c-8f93-398f9b6ecf29" containerName="payload-processing"
Sep 08 16:45:30.840759 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.840558    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7bb16b6-f188-4a0c-8f93-398f9b6ecf29" containerName="payload-processing"
Sep 08 16:45:30.840759 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.840566    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aa3f6b0-f4e3-49d6-9164-1151f2ced41a" containerName="revoke-keys"
Sep 08 16:45:30.840759 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.840571    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aa3f6b0-f4e3-49d6-9164-1151f2ced41a" containerName="revoke-keys"
Sep 08 16:45:30.840759 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.840621    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="25e24a89-d3b7-4413-95a0-035f73c78cb5" containerName="payload-processing"
Sep 08 16:45:30.840759 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.840627    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="67b7f626-d8af-4970-976b-f30ff734dba1" containerName="payload-pre-processing"
Sep 08 16:45:30.840759 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.840635    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aa3f6b0-f4e3-49d6-9164-1151f2ced41a" containerName="revoke-keys"
Sep 08 16:45:30.840759 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.840640    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a64a18e8-3dc0-48c3-8a09-189894e4b9ab" containerName="revoke-keys"
Sep 08 16:45:30.840759 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.840648    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7bb16b6-f188-4a0c-8f93-398f9b6ecf29" containerName="payload-processing"
Sep 08 16:45:30.843919 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.843904    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z"
Sep 08 16:45:30.846520 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.846498    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-773b1848\""
Sep 08 16:45:30.861106 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.861074    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z"]
Sep 08 16:45:30.949271 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.949233    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rqfg\" (UniqueName: \"kubernetes.io/projected/1089792f-a4ad-4003-909f-0d72904774ae-kube-api-access-5rqfg\") pod \"payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z\" (UID: \"1089792f-a4ad-4003-909f-0d72904774ae\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z"
Sep 08 16:45:30.949445 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:30.949322    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1089792f-a4ad-4003-909f-0d72904774ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z\" (UID: \"1089792f-a4ad-4003-909f-0d72904774ae\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z"
Sep 08 16:45:31.050230 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:31.050193    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1089792f-a4ad-4003-909f-0d72904774ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z\" (UID: \"1089792f-a4ad-4003-909f-0d72904774ae\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z"
Sep 08 16:45:31.050387 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:31.050283    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rqfg\" (UniqueName: \"kubernetes.io/projected/1089792f-a4ad-4003-909f-0d72904774ae-kube-api-access-5rqfg\") pod \"payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z\" (UID: \"1089792f-a4ad-4003-909f-0d72904774ae\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z"
Sep 08 16:45:31.050954 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:31.050935    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1089792f-a4ad-4003-909f-0d72904774ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z\" (UID: \"1089792f-a4ad-4003-909f-0d72904774ae\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z"
Sep 08 16:45:31.058748 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:31.058727    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rqfg\" (UniqueName: \"kubernetes.io/projected/1089792f-a4ad-4003-909f-0d72904774ae-kube-api-access-5rqfg\") pod \"payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z\" (UID: \"1089792f-a4ad-4003-909f-0d72904774ae\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z"
Sep 08 16:45:31.154632 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:31.154562    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z"
Sep 08 16:45:31.426387 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:31.426339    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z"]
Sep 08 16:45:31.428617 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:45:31.428592    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1089792f_a4ad_4003_909f_0d72904774ae.slice/crio-16c06648b303867939ddf7db9d30d9c054ed9d502de77cc66fc9f537184d69c6 WatchSource:0}: Error finding container 16c06648b303867939ddf7db9d30d9c054ed9d502de77cc66fc9f537184d69c6: Status 404 returned error can't find the container with id 16c06648b303867939ddf7db9d30d9c054ed9d502de77cc66fc9f537184d69c6
Sep 08 16:45:32.213322 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:32.213283    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z" event={"ID":"1089792f-a4ad-4003-909f-0d72904774ae","Type":"ContainerStarted","Data":"4aa9bd514285d681c5b702e88841d8c9514d8232f2b4373c47cb3e4288b5a0c9"}
Sep 08 16:45:32.213756 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:32.213342    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z" event={"ID":"1089792f-a4ad-4003-909f-0d72904774ae","Type":"ContainerStarted","Data":"16c06648b303867939ddf7db9d30d9c054ed9d502de77cc66fc9f537184d69c6"}
Sep 08 16:45:32.213756 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:32.213402    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z"
Sep 08 16:45:32.235686 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:32.235642    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z" podStartSLOduration=2.23562927 podStartE2EDuration="2.23562927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:45:32.23426615 +0000 UTC m=+916.397237597" watchObservedRunningTime="2026-09-08 16:45:32.23562927 +0000 UTC m=+916.398600717"
Sep 08 16:45:32.446805 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:32.446770    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf" podUID="5e58b226-2f45-46a0-ad65-f0eaebd08019" containerName="main" containerID="cri-o://734fb623b3109ad74099abef326eafdc194a5b80aff558aa7ef8cfacdddf589d" gracePeriod=45
Sep 08 16:45:32.699871 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:32.699822    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:45:32.766750 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:32.766720    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e58b226-2f45-46a0-ad65-f0eaebd08019-home\") pod \"5e58b226-2f45-46a0-ad65-f0eaebd08019\" (UID: \"5e58b226-2f45-46a0-ad65-f0eaebd08019\") "
Sep 08 16:45:32.766890 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:32.766867    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e58b226-2f45-46a0-ad65-f0eaebd08019-tmp-dir\") pod \"5e58b226-2f45-46a0-ad65-f0eaebd08019\" (UID: \"5e58b226-2f45-46a0-ad65-f0eaebd08019\") "
Sep 08 16:45:32.766890 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:32.766886    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e58b226-2f45-46a0-ad65-f0eaebd08019-model-cache\") pod \"5e58b226-2f45-46a0-ad65-f0eaebd08019\" (UID: \"5e58b226-2f45-46a0-ad65-f0eaebd08019\") "
Sep 08 16:45:32.766980 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:32.766930    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e58b226-2f45-46a0-ad65-f0eaebd08019-tls-certs\") pod \"5e58b226-2f45-46a0-ad65-f0eaebd08019\" (UID: \"5e58b226-2f45-46a0-ad65-f0eaebd08019\") "
Sep 08 16:45:32.766980 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:32.766955    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmjgg\" (UniqueName: \"kubernetes.io/projected/5e58b226-2f45-46a0-ad65-f0eaebd08019-kube-api-access-xmjgg\") pod \"5e58b226-2f45-46a0-ad65-f0eaebd08019\" (UID: \"5e58b226-2f45-46a0-ad65-f0eaebd08019\") "
Sep 08 16:45:32.767173 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:32.766973    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e58b226-2f45-46a0-ad65-f0eaebd08019-home" (OuterVolumeSpecName: "home") pod "5e58b226-2f45-46a0-ad65-f0eaebd08019" (UID: "5e58b226-2f45-46a0-ad65-f0eaebd08019"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:45:32.767173 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:32.766989    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e58b226-2f45-46a0-ad65-f0eaebd08019-dshm\") pod \"5e58b226-2f45-46a0-ad65-f0eaebd08019\" (UID: \"5e58b226-2f45-46a0-ad65-f0eaebd08019\") "
Sep 08 16:45:32.767173 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:32.767154    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e58b226-2f45-46a0-ad65-f0eaebd08019-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e58b226-2f45-46a0-ad65-f0eaebd08019" (UID: "5e58b226-2f45-46a0-ad65-f0eaebd08019"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:45:32.767287 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:32.767170    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e58b226-2f45-46a0-ad65-f0eaebd08019-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e58b226-2f45-46a0-ad65-f0eaebd08019" (UID: "5e58b226-2f45-46a0-ad65-f0eaebd08019"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:45:32.767287 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:32.767253    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e58b226-2f45-46a0-ad65-f0eaebd08019-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:32.767287 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:32.767271    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e58b226-2f45-46a0-ad65-f0eaebd08019-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:32.767287 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:32.767284    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e58b226-2f45-46a0-ad65-f0eaebd08019-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:32.769365 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:32.769340    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e58b226-2f45-46a0-ad65-f0eaebd08019-dshm" (OuterVolumeSpecName: "dshm") pod "5e58b226-2f45-46a0-ad65-f0eaebd08019" (UID: "5e58b226-2f45-46a0-ad65-f0eaebd08019"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:45:32.769469 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:32.769380    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e58b226-2f45-46a0-ad65-f0eaebd08019-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e58b226-2f45-46a0-ad65-f0eaebd08019" (UID: "5e58b226-2f45-46a0-ad65-f0eaebd08019"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:45:32.769469 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:32.769429    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e58b226-2f45-46a0-ad65-f0eaebd08019-kube-api-access-xmjgg" (OuterVolumeSpecName: "kube-api-access-xmjgg") pod "5e58b226-2f45-46a0-ad65-f0eaebd08019" (UID: "5e58b226-2f45-46a0-ad65-f0eaebd08019"). InnerVolumeSpecName "kube-api-access-xmjgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:45:32.867964 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:32.867892    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e58b226-2f45-46a0-ad65-f0eaebd08019-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:32.867964 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:32.867917    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmjgg\" (UniqueName: \"kubernetes.io/projected/5e58b226-2f45-46a0-ad65-f0eaebd08019-kube-api-access-xmjgg\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:32.867964 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:32.867926    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e58b226-2f45-46a0-ad65-f0eaebd08019-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:33.220478 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:33.220444    2557 generic.go:356] "Generic (PLEG): container finished" podID="5e58b226-2f45-46a0-ad65-f0eaebd08019" containerID="734fb623b3109ad74099abef326eafdc194a5b80aff558aa7ef8cfacdddf589d" exitCode=0
Sep 08 16:45:33.220876 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:33.220489    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"
Sep 08 16:45:33.220876 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:33.220529    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf" event={"ID":"5e58b226-2f45-46a0-ad65-f0eaebd08019","Type":"ContainerDied","Data":"734fb623b3109ad74099abef326eafdc194a5b80aff558aa7ef8cfacdddf589d"}
Sep 08 16:45:33.220876 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:33.220574    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf" event={"ID":"5e58b226-2f45-46a0-ad65-f0eaebd08019","Type":"ContainerDied","Data":"fefc44480610d1b1b15b063bc42df3460ee84f711198b1989c6a0a9dbe137e06"}
Sep 08 16:45:33.220876 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:33.220596    2557 scope.go:117] "RemoveContainer" containerID="734fb623b3109ad74099abef326eafdc194a5b80aff558aa7ef8cfacdddf589d"
Sep 08 16:45:33.232847 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:33.232819    2557 scope.go:117] "RemoveContainer" containerID="734fb623b3109ad74099abef326eafdc194a5b80aff558aa7ef8cfacdddf589d"
Sep 08 16:45:33.233108 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:45:33.233091    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"734fb623b3109ad74099abef326eafdc194a5b80aff558aa7ef8cfacdddf589d\": container with ID starting with 734fb623b3109ad74099abef326eafdc194a5b80aff558aa7ef8cfacdddf589d not found: ID does not exist" containerID="734fb623b3109ad74099abef326eafdc194a5b80aff558aa7ef8cfacdddf589d"
Sep 08 16:45:33.233157 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:33.233114    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"734fb623b3109ad74099abef326eafdc194a5b80aff558aa7ef8cfacdddf589d"} err="failed to get container status \"734fb623b3109ad74099abef326eafdc194a5b80aff558aa7ef8cfacdddf589d\": rpc error: code = NotFound desc = could not find container \"734fb623b3109ad74099abef326eafdc194a5b80aff558aa7ef8cfacdddf589d\": container with ID starting with 734fb623b3109ad74099abef326eafdc194a5b80aff558aa7ef8cfacdddf589d not found: ID does not exist"
Sep 08 16:45:33.247515 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:33.247492    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"]
Sep 08 16:45:33.253504 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:33.253473    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-1be25697-kserve-6555bf54f8-65wlf"]
Sep 08 16:45:33.943454 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:33.943387    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss" podUID="06040053-0109-456a-8c47-a7fdc2711b70" containerName="main" containerID="cri-o://590b4a25802b6d1c3a47b476f7559870a0e2dd8d98287c038af3d066409db95f" gracePeriod=45
Sep 08 16:45:34.196947 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.196891    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:45:34.229696 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.229667    2557 generic.go:356] "Generic (PLEG): container finished" podID="06040053-0109-456a-8c47-a7fdc2711b70" containerID="590b4a25802b6d1c3a47b476f7559870a0e2dd8d98287c038af3d066409db95f" exitCode=0
Sep 08 16:45:34.230123 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.229707    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"
Sep 08 16:45:34.230123 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.229733    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss" event={"ID":"06040053-0109-456a-8c47-a7fdc2711b70","Type":"ContainerDied","Data":"590b4a25802b6d1c3a47b476f7559870a0e2dd8d98287c038af3d066409db95f"}
Sep 08 16:45:34.230123 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.229761    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss" event={"ID":"06040053-0109-456a-8c47-a7fdc2711b70","Type":"ContainerDied","Data":"74179e253da68501df30f567434f8b81e6249bfd6be4db70362ce629882e353d"}
Sep 08 16:45:34.230123 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.229786    2557 scope.go:117] "RemoveContainer" containerID="590b4a25802b6d1c3a47b476f7559870a0e2dd8d98287c038af3d066409db95f"
Sep 08 16:45:34.238332 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.238316    2557 scope.go:117] "RemoveContainer" containerID="590b4a25802b6d1c3a47b476f7559870a0e2dd8d98287c038af3d066409db95f"
Sep 08 16:45:34.238608 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:45:34.238591    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"590b4a25802b6d1c3a47b476f7559870a0e2dd8d98287c038af3d066409db95f\": container with ID starting with 590b4a25802b6d1c3a47b476f7559870a0e2dd8d98287c038af3d066409db95f not found: ID does not exist" containerID="590b4a25802b6d1c3a47b476f7559870a0e2dd8d98287c038af3d066409db95f"
Sep 08 16:45:34.238674 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.238616    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"590b4a25802b6d1c3a47b476f7559870a0e2dd8d98287c038af3d066409db95f"} err="failed to get container status \"590b4a25802b6d1c3a47b476f7559870a0e2dd8d98287c038af3d066409db95f\": rpc error: code = NotFound desc = could not find container \"590b4a25802b6d1c3a47b476f7559870a0e2dd8d98287c038af3d066409db95f\": container with ID starting with 590b4a25802b6d1c3a47b476f7559870a0e2dd8d98287c038af3d066409db95f not found: ID does not exist"
Sep 08 16:45:34.280440 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.280412    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06040053-0109-456a-8c47-a7fdc2711b70-tmp-dir\") pod \"06040053-0109-456a-8c47-a7fdc2711b70\" (UID: \"06040053-0109-456a-8c47-a7fdc2711b70\") "
Sep 08 16:45:34.280547 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.280454    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06040053-0109-456a-8c47-a7fdc2711b70-home\") pod \"06040053-0109-456a-8c47-a7fdc2711b70\" (UID: \"06040053-0109-456a-8c47-a7fdc2711b70\") "
Sep 08 16:45:34.280547 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.280499    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrzrl\" (UniqueName: \"kubernetes.io/projected/06040053-0109-456a-8c47-a7fdc2711b70-kube-api-access-rrzrl\") pod \"06040053-0109-456a-8c47-a7fdc2711b70\" (UID: \"06040053-0109-456a-8c47-a7fdc2711b70\") "
Sep 08 16:45:34.280626 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.280567    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06040053-0109-456a-8c47-a7fdc2711b70-tls-certs\") pod \"06040053-0109-456a-8c47-a7fdc2711b70\" (UID: \"06040053-0109-456a-8c47-a7fdc2711b70\") "
Sep 08 16:45:34.280626 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.280599    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06040053-0109-456a-8c47-a7fdc2711b70-model-cache\") pod \"06040053-0109-456a-8c47-a7fdc2711b70\" (UID: \"06040053-0109-456a-8c47-a7fdc2711b70\") "
Sep 08 16:45:34.280714 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.280627    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06040053-0109-456a-8c47-a7fdc2711b70-dshm\") pod \"06040053-0109-456a-8c47-a7fdc2711b70\" (UID: \"06040053-0109-456a-8c47-a7fdc2711b70\") "
Sep 08 16:45:34.280714 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.280626    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06040053-0109-456a-8c47-a7fdc2711b70-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06040053-0109-456a-8c47-a7fdc2711b70" (UID: "06040053-0109-456a-8c47-a7fdc2711b70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:45:34.280714 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.280693    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06040053-0109-456a-8c47-a7fdc2711b70-home" (OuterVolumeSpecName: "home") pod "06040053-0109-456a-8c47-a7fdc2711b70" (UID: "06040053-0109-456a-8c47-a7fdc2711b70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:45:34.280903 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.280885    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06040053-0109-456a-8c47-a7fdc2711b70-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:34.280975 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.280930    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06040053-0109-456a-8c47-a7fdc2711b70-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:34.281076 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.281050    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06040053-0109-456a-8c47-a7fdc2711b70-model-cache" (OuterVolumeSpecName: "model-cache") pod "06040053-0109-456a-8c47-a7fdc2711b70" (UID: "06040053-0109-456a-8c47-a7fdc2711b70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:45:34.282874 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.282818    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06040053-0109-456a-8c47-a7fdc2711b70-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06040053-0109-456a-8c47-a7fdc2711b70" (UID: "06040053-0109-456a-8c47-a7fdc2711b70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:45:34.282981 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.282927    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06040053-0109-456a-8c47-a7fdc2711b70-kube-api-access-rrzrl" (OuterVolumeSpecName: "kube-api-access-rrzrl") pod "06040053-0109-456a-8c47-a7fdc2711b70" (UID: "06040053-0109-456a-8c47-a7fdc2711b70"). InnerVolumeSpecName "kube-api-access-rrzrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:45:34.283055 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.283038    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06040053-0109-456a-8c47-a7fdc2711b70-dshm" (OuterVolumeSpecName: "dshm") pod "06040053-0109-456a-8c47-a7fdc2711b70" (UID: "06040053-0109-456a-8c47-a7fdc2711b70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:45:34.345034 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.345010    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e58b226-2f45-46a0-ad65-f0eaebd08019" path="/var/lib/kubelet/pods/5e58b226-2f45-46a0-ad65-f0eaebd08019/volumes"
Sep 08 16:45:34.381881 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.381856    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06040053-0109-456a-8c47-a7fdc2711b70-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:34.381881 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.381881    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06040053-0109-456a-8c47-a7fdc2711b70-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:34.382004 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.381894    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrzrl\" (UniqueName: \"kubernetes.io/projected/06040053-0109-456a-8c47-a7fdc2711b70-kube-api-access-rrzrl\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:34.382004 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.381907    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06040053-0109-456a-8c47-a7fdc2711b70-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:34.545566 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.545502    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"]
Sep 08 16:45:34.551735 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.551713    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-4cea93/autores-model-4cea93-kserve-75c49ccfc5-xspss"]
Sep 08 16:45:34.737216 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.737184    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf" podUID="4cce2c68-26b9-49b5-920e-021bc699ee5a" containerName="main" containerID="cri-o://f597f8e522081d457e118cb89fc8ea083cce18c6bb72b01a3db4395ec3459c18" gracePeriod=45
Sep 08 16:45:34.992552 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:34.992529    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:45:35.087601 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.087565    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cce2c68-26b9-49b5-920e-021bc699ee5a-tls-certs\") pod \"4cce2c68-26b9-49b5-920e-021bc699ee5a\" (UID: \"4cce2c68-26b9-49b5-920e-021bc699ee5a\") "
Sep 08 16:45:35.087773 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.087610    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cce2c68-26b9-49b5-920e-021bc699ee5a-model-cache\") pod \"4cce2c68-26b9-49b5-920e-021bc699ee5a\" (UID: \"4cce2c68-26b9-49b5-920e-021bc699ee5a\") "
Sep 08 16:45:35.087773 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.087638    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cce2c68-26b9-49b5-920e-021bc699ee5a-home\") pod \"4cce2c68-26b9-49b5-920e-021bc699ee5a\" (UID: \"4cce2c68-26b9-49b5-920e-021bc699ee5a\") "
Sep 08 16:45:35.087773 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.087675    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wp2sh\" (UniqueName: \"kubernetes.io/projected/4cce2c68-26b9-49b5-920e-021bc699ee5a-kube-api-access-wp2sh\") pod \"4cce2c68-26b9-49b5-920e-021bc699ee5a\" (UID: \"4cce2c68-26b9-49b5-920e-021bc699ee5a\") "
Sep 08 16:45:35.087773 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.087716    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cce2c68-26b9-49b5-920e-021bc699ee5a-dshm\") pod \"4cce2c68-26b9-49b5-920e-021bc699ee5a\" (UID: \"4cce2c68-26b9-49b5-920e-021bc699ee5a\") "
Sep 08 16:45:35.088020 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.087804    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cce2c68-26b9-49b5-920e-021bc699ee5a-tmp-dir\") pod \"4cce2c68-26b9-49b5-920e-021bc699ee5a\" (UID: \"4cce2c68-26b9-49b5-920e-021bc699ee5a\") "
Sep 08 16:45:35.088020 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.087867    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cce2c68-26b9-49b5-920e-021bc699ee5a-model-cache" (OuterVolumeSpecName: "model-cache") pod "4cce2c68-26b9-49b5-920e-021bc699ee5a" (UID: "4cce2c68-26b9-49b5-920e-021bc699ee5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:45:35.088020 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.087911    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cce2c68-26b9-49b5-920e-021bc699ee5a-home" (OuterVolumeSpecName: "home") pod "4cce2c68-26b9-49b5-920e-021bc699ee5a" (UID: "4cce2c68-26b9-49b5-920e-021bc699ee5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:45:35.088196 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.088106    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cce2c68-26b9-49b5-920e-021bc699ee5a-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:35.088196 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.088111    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cce2c68-26b9-49b5-920e-021bc699ee5a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4cce2c68-26b9-49b5-920e-021bc699ee5a" (UID: "4cce2c68-26b9-49b5-920e-021bc699ee5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:45:35.088196 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.088126    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cce2c68-26b9-49b5-920e-021bc699ee5a-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:35.090041 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.090016    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cce2c68-26b9-49b5-920e-021bc699ee5a-dshm" (OuterVolumeSpecName: "dshm") pod "4cce2c68-26b9-49b5-920e-021bc699ee5a" (UID: "4cce2c68-26b9-49b5-920e-021bc699ee5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:45:35.090163 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.090036    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cce2c68-26b9-49b5-920e-021bc699ee5a-kube-api-access-wp2sh" (OuterVolumeSpecName: "kube-api-access-wp2sh") pod "4cce2c68-26b9-49b5-920e-021bc699ee5a" (UID: "4cce2c68-26b9-49b5-920e-021bc699ee5a"). InnerVolumeSpecName "kube-api-access-wp2sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:45:35.090163 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.090137    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cce2c68-26b9-49b5-920e-021bc699ee5a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4cce2c68-26b9-49b5-920e-021bc699ee5a" (UID: "4cce2c68-26b9-49b5-920e-021bc699ee5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:45:35.188801 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.188771    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp2sh\" (UniqueName: \"kubernetes.io/projected/4cce2c68-26b9-49b5-920e-021bc699ee5a-kube-api-access-wp2sh\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:35.188801 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.188800    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cce2c68-26b9-49b5-920e-021bc699ee5a-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:35.188995 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.188816    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cce2c68-26b9-49b5-920e-021bc699ee5a-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:35.188995 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.188828    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cce2c68-26b9-49b5-920e-021bc699ee5a-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:35.237160 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.237129    2557 generic.go:356] "Generic (PLEG): container finished" podID="4cce2c68-26b9-49b5-920e-021bc699ee5a" containerID="f597f8e522081d457e118cb89fc8ea083cce18c6bb72b01a3db4395ec3459c18" exitCode=0
Sep 08 16:45:35.237499 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.237176    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"
Sep 08 16:45:35.237499 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.237196    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf" event={"ID":"4cce2c68-26b9-49b5-920e-021bc699ee5a","Type":"ContainerDied","Data":"f597f8e522081d457e118cb89fc8ea083cce18c6bb72b01a3db4395ec3459c18"}
Sep 08 16:45:35.237499 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.237233    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf" event={"ID":"4cce2c68-26b9-49b5-920e-021bc699ee5a","Type":"ContainerDied","Data":"fd63f2b7e9e534b00990006ff1b08771f5ba3e45128b1561d2d5061b0a1b9ee3"}
Sep 08 16:45:35.237499 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.237256    2557 scope.go:117] "RemoveContainer" containerID="f597f8e522081d457e118cb89fc8ea083cce18c6bb72b01a3db4395ec3459c18"
Sep 08 16:45:35.246149 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.246132    2557 scope.go:117] "RemoveContainer" containerID="f597f8e522081d457e118cb89fc8ea083cce18c6bb72b01a3db4395ec3459c18"
Sep 08 16:45:35.246380 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:45:35.246366    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f597f8e522081d457e118cb89fc8ea083cce18c6bb72b01a3db4395ec3459c18\": container with ID starting with f597f8e522081d457e118cb89fc8ea083cce18c6bb72b01a3db4395ec3459c18 not found: ID does not exist" containerID="f597f8e522081d457e118cb89fc8ea083cce18c6bb72b01a3db4395ec3459c18"
Sep 08 16:45:35.246439 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.246386    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f597f8e522081d457e118cb89fc8ea083cce18c6bb72b01a3db4395ec3459c18"} err="failed to get container status \"f597f8e522081d457e118cb89fc8ea083cce18c6bb72b01a3db4395ec3459c18\": rpc error: code = NotFound desc = could not find container \"f597f8e522081d457e118cb89fc8ea083cce18c6bb72b01a3db4395ec3459c18\": container with ID starting with f597f8e522081d457e118cb89fc8ea083cce18c6bb72b01a3db4395ec3459c18 not found: ID does not exist"
Sep 08 16:45:35.259480 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.259461    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"]
Sep 08 16:45:35.265227 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:35.265206    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-1be25697-kserve-867b46f6f6-hv8xf"]
Sep 08 16:45:36.345822 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:36.345790    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06040053-0109-456a-8c47-a7fdc2711b70" path="/var/lib/kubelet/pods/06040053-0109-456a-8c47-a7fdc2711b70/volumes"
Sep 08 16:45:36.346189 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:36.346170    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cce2c68-26b9-49b5-920e-021bc699ee5a" path="/var/lib/kubelet/pods/4cce2c68-26b9-49b5-920e-021bc699ee5a/volumes"
Sep 08 16:45:36.536147 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:36.536114    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:45:43.222936 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.222904    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z"
Sep 08 16:45:43.641603 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.641513    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn"]
Sep 08 16:45:43.641979 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.641960    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e58b226-2f45-46a0-ad65-f0eaebd08019" containerName="main"
Sep 08 16:45:43.642118 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.641982    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e58b226-2f45-46a0-ad65-f0eaebd08019" containerName="main"
Sep 08 16:45:43.642118 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.641996    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cce2c68-26b9-49b5-920e-021bc699ee5a" containerName="main"
Sep 08 16:45:43.642118 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.642003    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cce2c68-26b9-49b5-920e-021bc699ee5a" containerName="main"
Sep 08 16:45:43.642118 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.642064    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06040053-0109-456a-8c47-a7fdc2711b70" containerName="main"
Sep 08 16:45:43.642118 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.642073    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="06040053-0109-456a-8c47-a7fdc2711b70" containerName="main"
Sep 08 16:45:43.642351 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.642163    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="06040053-0109-456a-8c47-a7fdc2711b70" containerName="main"
Sep 08 16:45:43.642351 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.642177    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e58b226-2f45-46a0-ad65-f0eaebd08019" containerName="main"
Sep 08 16:45:43.642351 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.642189    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cce2c68-26b9-49b5-920e-021bc699ee5a" containerName="main"
Sep 08 16:45:43.645096 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.645075    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn"
Sep 08 16:45:43.647642 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.647623    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-fae677\""
Sep 08 16:45:43.647723 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.647624    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-fae677-dockercfg-hgvwl\""
Sep 08 16:45:43.652712 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.652696    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn"]
Sep 08 16:45:43.751216 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.751190    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq"]
Sep 08 16:45:43.754522 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.754504    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq"
Sep 08 16:45:43.758297 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.758278    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97br9\" (UniqueName: \"kubernetes.io/projected/d8aaccba-6909-4458-88f2-768bfc829d9b-kube-api-access-97br9\") pod \"payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn\" (UID: \"d8aaccba-6909-4458-88f2-768bfc829d9b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn"
Sep 08 16:45:43.758377 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.758357    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8aaccba-6909-4458-88f2-768bfc829d9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn\" (UID: \"d8aaccba-6909-4458-88f2-768bfc829d9b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn"
Sep 08 16:45:43.777768 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.777748    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq"]
Sep 08 16:45:43.858706 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.858671    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8aaccba-6909-4458-88f2-768bfc829d9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn\" (UID: \"d8aaccba-6909-4458-88f2-768bfc829d9b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn"
Sep 08 16:45:43.858706 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.858710    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97br9\" (UniqueName: \"kubernetes.io/projected/d8aaccba-6909-4458-88f2-768bfc829d9b-kube-api-access-97br9\") pod \"payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn\" (UID: \"d8aaccba-6909-4458-88f2-768bfc829d9b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn"
Sep 08 16:45:43.858956 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.858736    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e45b54e-f8e5-45ad-889e-2b7de92c9f0d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq\" (UID: \"5e45b54e-f8e5-45ad-889e-2b7de92c9f0d\") " pod="openshift-ingress/payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq"
Sep 08 16:45:43.858956 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.858769    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcm8w\" (UniqueName: \"kubernetes.io/projected/5e45b54e-f8e5-45ad-889e-2b7de92c9f0d-kube-api-access-kcm8w\") pod \"payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq\" (UID: \"5e45b54e-f8e5-45ad-889e-2b7de92c9f0d\") " pod="openshift-ingress/payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq"
Sep 08 16:45:43.859408 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.859384    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8aaccba-6909-4458-88f2-768bfc829d9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn\" (UID: \"d8aaccba-6909-4458-88f2-768bfc829d9b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn"
Sep 08 16:45:43.867644 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.867620    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97br9\" (UniqueName: \"kubernetes.io/projected/d8aaccba-6909-4458-88f2-768bfc829d9b-kube-api-access-97br9\") pod \"payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn\" (UID: \"d8aaccba-6909-4458-88f2-768bfc829d9b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn"
Sep 08 16:45:43.955941 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.955912    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn"
Sep 08 16:45:43.960000 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.959981    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e45b54e-f8e5-45ad-889e-2b7de92c9f0d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq\" (UID: \"5e45b54e-f8e5-45ad-889e-2b7de92c9f0d\") " pod="openshift-ingress/payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq"
Sep 08 16:45:43.960059 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.960015    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcm8w\" (UniqueName: \"kubernetes.io/projected/5e45b54e-f8e5-45ad-889e-2b7de92c9f0d-kube-api-access-kcm8w\") pod \"payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq\" (UID: \"5e45b54e-f8e5-45ad-889e-2b7de92c9f0d\") " pod="openshift-ingress/payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq"
Sep 08 16:45:43.960665 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.960647    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e45b54e-f8e5-45ad-889e-2b7de92c9f0d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq\" (UID: \"5e45b54e-f8e5-45ad-889e-2b7de92c9f0d\") " pod="openshift-ingress/payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq"
Sep 08 16:45:43.968279 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:43.968254    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcm8w\" (UniqueName: \"kubernetes.io/projected/5e45b54e-f8e5-45ad-889e-2b7de92c9f0d-kube-api-access-kcm8w\") pod \"payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq\" (UID: \"5e45b54e-f8e5-45ad-889e-2b7de92c9f0d\") " pod="openshift-ingress/payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq"
Sep 08 16:45:44.064193 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:44.064164    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq"
Sep 08 16:45:44.229261 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:44.229233    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn"]
Sep 08 16:45:44.231015 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:45:44.230988    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8aaccba_6909_4458_88f2_768bfc829d9b.slice/crio-d1273758bdfe930b403d748f87353e445676951b2e6cc6f1c8a1e62a0c0e41b1 WatchSource:0}: Error finding container d1273758bdfe930b403d748f87353e445676951b2e6cc6f1c8a1e62a0c0e41b1: Status 404 returned error can't find the container with id d1273758bdfe930b403d748f87353e445676951b2e6cc6f1c8a1e62a0c0e41b1
Sep 08 16:45:44.294298 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:44.294274    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn" event={"ID":"d8aaccba-6909-4458-88f2-768bfc829d9b","Type":"ContainerStarted","Data":"045f7d922c9b25b952e2ecdbd753b712f437e22fbd26312308c30230164474ed"}
Sep 08 16:45:44.294437 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:44.294331    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn" event={"ID":"d8aaccba-6909-4458-88f2-768bfc829d9b","Type":"ContainerStarted","Data":"d1273758bdfe930b403d748f87353e445676951b2e6cc6f1c8a1e62a0c0e41b1"}
Sep 08 16:45:44.294437 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:44.294391    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn"
Sep 08 16:45:44.309959 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:44.309912    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn" podStartSLOduration=1.309896224 podStartE2EDuration="1.309896224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:45:44.308889575 +0000 UTC m=+928.471861044" watchObservedRunningTime="2026-09-08 16:45:44.309896224 +0000 UTC m=+928.472867672"
Sep 08 16:45:44.334393 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:44.334374    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq"]
Sep 08 16:45:44.335806 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:45:44.335774    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e45b54e_f8e5_45ad_889e_2b7de92c9f0d.slice/crio-0eb499b06b7c4798b16c069745e9f9b7f641223030c385e41554b9b8959f617e WatchSource:0}: Error finding container 0eb499b06b7c4798b16c069745e9f9b7f641223030c385e41554b9b8959f617e: Status 404 returned error can't find the container with id 0eb499b06b7c4798b16c069745e9f9b7f641223030c385e41554b9b8959f617e
Sep 08 16:45:45.301822 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:45.301778    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq" event={"ID":"5e45b54e-f8e5-45ad-889e-2b7de92c9f0d","Type":"ContainerStarted","Data":"24e6d57ebc7504eecd75223a94f608198eb008ce9119d53f20b8b6a71f0f8ed2"}
Sep 08 16:45:45.302283 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:45.301854    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq" event={"ID":"5e45b54e-f8e5-45ad-889e-2b7de92c9f0d","Type":"ContainerStarted","Data":"0eb499b06b7c4798b16c069745e9f9b7f641223030c385e41554b9b8959f617e"}
Sep 08 16:45:45.302283 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:45.302006    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq"
Sep 08 16:45:45.319200 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:45.319154    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq" podStartSLOduration=2.3191402 podStartE2EDuration="2.3191402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:45:45.318581262 +0000 UTC m=+929.481552712" watchObservedRunningTime="2026-09-08 16:45:45.3191402 +0000 UTC m=+929.482111677"
Sep 08 16:45:52.931924 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:52.931888    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:45:53.485773 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:53.485736    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-773b1848-smtp2"]
Sep 08 16:45:53.490383 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:53.490353    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-773b1848-smtp2"
Sep 08 16:45:53.492874 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:53.492855    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:45:53.493005 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:53.492875    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-886jf\""
Sep 08 16:45:53.496931 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:53.496902    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-773b1848-smtp2"]
Sep 08 16:45:53.643781 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:53.643748    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55a769fa-1d64-421d-8336-215e906d9195-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-773b1848-smtp2\" (UID: \"55a769fa-1d64-421d-8336-215e906d9195\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-773b1848-smtp2"
Sep 08 16:45:53.744730 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:53.744655    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55a769fa-1d64-421d-8336-215e906d9195-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-773b1848-smtp2\" (UID: \"55a769fa-1d64-421d-8336-215e906d9195\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-773b1848-smtp2"
Sep 08 16:45:53.745268 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:53.745251    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55a769fa-1d64-421d-8336-215e906d9195-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-773b1848-smtp2\" (UID: \"55a769fa-1d64-421d-8336-215e906d9195\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-773b1848-smtp2"
Sep 08 16:45:53.804483 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:53.804454    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-773b1848-smtp2"
Sep 08 16:45:54.073551 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:54.073448    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-773b1848-smtp2"]
Sep 08 16:45:54.076270 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:45:54.076242    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a769fa_1d64_421d_8336_215e906d9195.slice/crio-b4d9ab631601b42b83f7f4c2b0ceb3c384d5daa75a24ec29598968908cbec7f8 WatchSource:0}: Error finding container b4d9ab631601b42b83f7f4c2b0ceb3c384d5daa75a24ec29598968908cbec7f8: Status 404 returned error can't find the container with id b4d9ab631601b42b83f7f4c2b0ceb3c384d5daa75a24ec29598968908cbec7f8
Sep 08 16:45:54.349750 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:54.349679    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-773b1848-smtp2" event={"ID":"55a769fa-1d64-421d-8336-215e906d9195","Type":"ContainerStarted","Data":"b4d9ab631601b42b83f7f4c2b0ceb3c384d5daa75a24ec29598968908cbec7f8"}
Sep 08 16:45:55.303595 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:55.303565    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn"
Sep 08 16:45:55.356174 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:55.356145    2557 generic.go:356] "Generic (PLEG): container finished" podID="55a769fa-1d64-421d-8336-215e906d9195" containerID="8414a7c1f79a0bf3360933e861173972e565e64a8dd5111c78d136f087441fe1" exitCode=0
Sep 08 16:45:55.356330 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:55.356229    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-773b1848-smtp2" event={"ID":"55a769fa-1d64-421d-8336-215e906d9195","Type":"ContainerDied","Data":"8414a7c1f79a0bf3360933e861173972e565e64a8dd5111c78d136f087441fe1"}
Sep 08 16:45:56.310013 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:56.309983    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq"
Sep 08 16:45:56.503870 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:56.503826    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-773b1848-smtp2"
Sep 08 16:45:56.568847 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:56.568765    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55a769fa-1d64-421d-8336-215e906d9195-maas-api-service-ca\") pod \"55a769fa-1d64-421d-8336-215e906d9195\" (UID: \"55a769fa-1d64-421d-8336-215e906d9195\") "
Sep 08 16:45:56.569131 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:56.569111    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a769fa-1d64-421d-8336-215e906d9195-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "55a769fa-1d64-421d-8336-215e906d9195" (UID: "55a769fa-1d64-421d-8336-215e906d9195"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:45:56.669312 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:56.669283    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55a769fa-1d64-421d-8336-215e906d9195-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:45:57.368209 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:57.368180    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-773b1848-smtp2"
Sep 08 16:45:57.368209 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:57.368210    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-773b1848-smtp2" event={"ID":"55a769fa-1d64-421d-8336-215e906d9195","Type":"ContainerDied","Data":"b4d9ab631601b42b83f7f4c2b0ceb3c384d5daa75a24ec29598968908cbec7f8"}
Sep 08 16:45:57.368652 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:45:57.368256    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4d9ab631601b42b83f7f4c2b0ceb3c384d5daa75a24ec29598968908cbec7f8"
Sep 08 16:46:00.691022 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:00.690991    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z"]
Sep 08 16:46:00.691537 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:00.691189    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z" podUID="1089792f-a4ad-4003-909f-0d72904774ae" containerName="payload-pre-processing" containerID="cri-o://4aa9bd514285d681c5b702e88841d8c9514d8232f2b4373c47cb3e4288b5a0c9" gracePeriod=30
Sep 08 16:46:00.872280 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:00.872204    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-773b1848-smtp2"]
Sep 08 16:46:00.875493 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:00.875467    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-773b1848-smtp2"]
Sep 08 16:46:00.934095 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:00.934070    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z"
Sep 08 16:46:01.004607 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:01.004538    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rqfg\" (UniqueName: \"kubernetes.io/projected/1089792f-a4ad-4003-909f-0d72904774ae-kube-api-access-5rqfg\") pod \"1089792f-a4ad-4003-909f-0d72904774ae\" (UID: \"1089792f-a4ad-4003-909f-0d72904774ae\") "
Sep 08 16:46:01.004607 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:01.004583    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1089792f-a4ad-4003-909f-0d72904774ae-plugins-config-volume\") pod \"1089792f-a4ad-4003-909f-0d72904774ae\" (UID: \"1089792f-a4ad-4003-909f-0d72904774ae\") "
Sep 08 16:46:01.005049 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:01.005025    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1089792f-a4ad-4003-909f-0d72904774ae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1089792f-a4ad-4003-909f-0d72904774ae" (UID: "1089792f-a4ad-4003-909f-0d72904774ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:46:01.006794 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:01.006758    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1089792f-a4ad-4003-909f-0d72904774ae-kube-api-access-5rqfg" (OuterVolumeSpecName: "kube-api-access-5rqfg") pod "1089792f-a4ad-4003-909f-0d72904774ae" (UID: "1089792f-a4ad-4003-909f-0d72904774ae"). InnerVolumeSpecName "kube-api-access-5rqfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:46:01.105517 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:01.105484    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rqfg\" (UniqueName: \"kubernetes.io/projected/1089792f-a4ad-4003-909f-0d72904774ae-kube-api-access-5rqfg\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:46:01.105517 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:01.105514    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1089792f-a4ad-4003-909f-0d72904774ae-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:46:01.390923 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:01.390823    2557 generic.go:356] "Generic (PLEG): container finished" podID="1089792f-a4ad-4003-909f-0d72904774ae" containerID="4aa9bd514285d681c5b702e88841d8c9514d8232f2b4373c47cb3e4288b5a0c9" exitCode=0
Sep 08 16:46:01.390923 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:01.390882    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z"
Sep 08 16:46:01.391104 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:01.390936    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z" event={"ID":"1089792f-a4ad-4003-909f-0d72904774ae","Type":"ContainerDied","Data":"4aa9bd514285d681c5b702e88841d8c9514d8232f2b4373c47cb3e4288b5a0c9"}
Sep 08 16:46:01.391104 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:01.390982    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z" event={"ID":"1089792f-a4ad-4003-909f-0d72904774ae","Type":"ContainerDied","Data":"16c06648b303867939ddf7db9d30d9c054ed9d502de77cc66fc9f537184d69c6"}
Sep 08 16:46:01.391104 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:01.391009    2557 scope.go:117] "RemoveContainer" containerID="4aa9bd514285d681c5b702e88841d8c9514d8232f2b4373c47cb3e4288b5a0c9"
Sep 08 16:46:01.400720 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:01.400703    2557 scope.go:117] "RemoveContainer" containerID="4aa9bd514285d681c5b702e88841d8c9514d8232f2b4373c47cb3e4288b5a0c9"
Sep 08 16:46:01.401067 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:46:01.401043    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa9bd514285d681c5b702e88841d8c9514d8232f2b4373c47cb3e4288b5a0c9\": container with ID starting with 4aa9bd514285d681c5b702e88841d8c9514d8232f2b4373c47cb3e4288b5a0c9 not found: ID does not exist" containerID="4aa9bd514285d681c5b702e88841d8c9514d8232f2b4373c47cb3e4288b5a0c9"
Sep 08 16:46:01.401141 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:01.401077    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa9bd514285d681c5b702e88841d8c9514d8232f2b4373c47cb3e4288b5a0c9"} err="failed to get container status \"4aa9bd514285d681c5b702e88841d8c9514d8232f2b4373c47cb3e4288b5a0c9\": rpc error: code = NotFound desc = could not find container \"4aa9bd514285d681c5b702e88841d8c9514d8232f2b4373c47cb3e4288b5a0c9\": container with ID starting with 4aa9bd514285d681c5b702e88841d8c9514d8232f2b4373c47cb3e4288b5a0c9 not found: ID does not exist"
Sep 08 16:46:01.414594 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:01.414565    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z"]
Sep 08 16:46:01.416682 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:01.416658    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-773b1848-556ff8b49d-j9t7z"]
Sep 08 16:46:02.346321 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:02.346279    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1089792f-a4ad-4003-909f-0d72904774ae" path="/var/lib/kubelet/pods/1089792f-a4ad-4003-909f-0d72904774ae/volumes"
Sep 08 16:46:02.346808 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:02.346749    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55a769fa-1d64-421d-8336-215e906d9195" path="/var/lib/kubelet/pods/55a769fa-1d64-421d-8336-215e906d9195/volumes"
Sep 08 16:46:06.840014 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:06.839982    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:46:09.708518 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.708476    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"]
Sep 08 16:46:09.708896 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.708843    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1089792f-a4ad-4003-909f-0d72904774ae" containerName="payload-pre-processing"
Sep 08 16:46:09.708896 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.708857    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1089792f-a4ad-4003-909f-0d72904774ae" containerName="payload-pre-processing"
Sep 08 16:46:09.708896 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.708871    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55a769fa-1d64-421d-8336-215e906d9195" containerName="revoke-keys"
Sep 08 16:46:09.708896 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.708876    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="55a769fa-1d64-421d-8336-215e906d9195" containerName="revoke-keys"
Sep 08 16:46:09.709040 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.708929    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="55a769fa-1d64-421d-8336-215e906d9195" containerName="revoke-keys"
Sep 08 16:46:09.709040 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.708938    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1089792f-a4ad-4003-909f-0d72904774ae" containerName="payload-pre-processing"
Sep 08 16:46:09.713299 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.713269    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:46:09.718915 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.718891    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-43a905\"/\"kube-root-ca.crt\""
Sep 08 16:46:09.719056 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.718947    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-43a905\"/\"default-dockercfg-j9kzl\""
Sep 08 16:46:09.719056 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.718890    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-43a905\"/\"test-model-43a905-kserve-self-signed-certs\""
Sep 08 16:46:09.720546 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.720525    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-43a905\"/\"openshift-service-ca.crt\""
Sep 08 16:46:09.744121 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.744098    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"]
Sep 08 16:46:09.780762 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.780732    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cd532f1-a543-40d4-a6fa-cee22936725a-home\") pod \"test-model-43a905-kserve-5bd5f577df-6lq9p\" (UID: \"2cd532f1-a543-40d4-a6fa-cee22936725a\") " pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:46:09.780762 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.780760    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cd532f1-a543-40d4-a6fa-cee22936725a-dshm\") pod \"test-model-43a905-kserve-5bd5f577df-6lq9p\" (UID: \"2cd532f1-a543-40d4-a6fa-cee22936725a\") " pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:46:09.780959 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.780775    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cd532f1-a543-40d4-a6fa-cee22936725a-tmp-dir\") pod \"test-model-43a905-kserve-5bd5f577df-6lq9p\" (UID: \"2cd532f1-a543-40d4-a6fa-cee22936725a\") " pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:46:09.780959 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.780826    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cd532f1-a543-40d4-a6fa-cee22936725a-tls-certs\") pod \"test-model-43a905-kserve-5bd5f577df-6lq9p\" (UID: \"2cd532f1-a543-40d4-a6fa-cee22936725a\") " pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:46:09.780959 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.780935    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cd532f1-a543-40d4-a6fa-cee22936725a-model-cache\") pod \"test-model-43a905-kserve-5bd5f577df-6lq9p\" (UID: \"2cd532f1-a543-40d4-a6fa-cee22936725a\") " pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:46:09.781077 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.780967    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtzlt\" (UniqueName: \"kubernetes.io/projected/2cd532f1-a543-40d4-a6fa-cee22936725a-kube-api-access-dtzlt\") pod \"test-model-43a905-kserve-5bd5f577df-6lq9p\" (UID: \"2cd532f1-a543-40d4-a6fa-cee22936725a\") " pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:46:09.881663 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.881635    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtzlt\" (UniqueName: \"kubernetes.io/projected/2cd532f1-a543-40d4-a6fa-cee22936725a-kube-api-access-dtzlt\") pod \"test-model-43a905-kserve-5bd5f577df-6lq9p\" (UID: \"2cd532f1-a543-40d4-a6fa-cee22936725a\") " pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:46:09.881819 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.881673    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cd532f1-a543-40d4-a6fa-cee22936725a-home\") pod \"test-model-43a905-kserve-5bd5f577df-6lq9p\" (UID: \"2cd532f1-a543-40d4-a6fa-cee22936725a\") " pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:46:09.881819 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.881687    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cd532f1-a543-40d4-a6fa-cee22936725a-dshm\") pod \"test-model-43a905-kserve-5bd5f577df-6lq9p\" (UID: \"2cd532f1-a543-40d4-a6fa-cee22936725a\") " pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:46:09.881819 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.881705    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cd532f1-a543-40d4-a6fa-cee22936725a-tmp-dir\") pod \"test-model-43a905-kserve-5bd5f577df-6lq9p\" (UID: \"2cd532f1-a543-40d4-a6fa-cee22936725a\") " pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:46:09.881819 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.881743    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cd532f1-a543-40d4-a6fa-cee22936725a-tls-certs\") pod \"test-model-43a905-kserve-5bd5f577df-6lq9p\" (UID: \"2cd532f1-a543-40d4-a6fa-cee22936725a\") " pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:46:09.881819 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.881801    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cd532f1-a543-40d4-a6fa-cee22936725a-model-cache\") pod \"test-model-43a905-kserve-5bd5f577df-6lq9p\" (UID: \"2cd532f1-a543-40d4-a6fa-cee22936725a\") " pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:46:09.882140 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.882117    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cd532f1-a543-40d4-a6fa-cee22936725a-tmp-dir\") pod \"test-model-43a905-kserve-5bd5f577df-6lq9p\" (UID: \"2cd532f1-a543-40d4-a6fa-cee22936725a\") " pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:46:09.882140 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.882133    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cd532f1-a543-40d4-a6fa-cee22936725a-home\") pod \"test-model-43a905-kserve-5bd5f577df-6lq9p\" (UID: \"2cd532f1-a543-40d4-a6fa-cee22936725a\") " pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:46:09.882236 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.882221    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cd532f1-a543-40d4-a6fa-cee22936725a-model-cache\") pod \"test-model-43a905-kserve-5bd5f577df-6lq9p\" (UID: \"2cd532f1-a543-40d4-a6fa-cee22936725a\") " pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:46:09.884192 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.884172    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cd532f1-a543-40d4-a6fa-cee22936725a-dshm\") pod \"test-model-43a905-kserve-5bd5f577df-6lq9p\" (UID: \"2cd532f1-a543-40d4-a6fa-cee22936725a\") " pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:46:09.884435 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.884418    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cd532f1-a543-40d4-a6fa-cee22936725a-tls-certs\") pod \"test-model-43a905-kserve-5bd5f577df-6lq9p\" (UID: \"2cd532f1-a543-40d4-a6fa-cee22936725a\") " pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:46:09.895465 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:09.895436    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtzlt\" (UniqueName: \"kubernetes.io/projected/2cd532f1-a543-40d4-a6fa-cee22936725a-kube-api-access-dtzlt\") pod \"test-model-43a905-kserve-5bd5f577df-6lq9p\" (UID: \"2cd532f1-a543-40d4-a6fa-cee22936725a\") " pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:46:10.024808 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:10.024731    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:46:10.300749 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:10.300726    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"]
Sep 08 16:46:10.302911 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:46:10.302884    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cd532f1_a543_40d4_a6fa_cee22936725a.slice/crio-9d15ead713890b89887bc582ce49e587edcd3933f15c23343e7dc29cd4374d3e WatchSource:0}: Error finding container 9d15ead713890b89887bc582ce49e587edcd3933f15c23343e7dc29cd4374d3e: Status 404 returned error can't find the container with id 9d15ead713890b89887bc582ce49e587edcd3933f15c23343e7dc29cd4374d3e
Sep 08 16:46:10.443772 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:10.443724    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p" event={"ID":"2cd532f1-a543-40d4-a6fa-cee22936725a","Type":"ContainerStarted","Data":"a3b031f803e7bf38884cbe7b857d5b23d9d4605bec62dc46a174b87df17abd0d"}
Sep 08 16:46:10.443772 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:10.443775    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p" event={"ID":"2cd532f1-a543-40d4-a6fa-cee22936725a","Type":"ContainerStarted","Data":"9d15ead713890b89887bc582ce49e587edcd3933f15c23343e7dc29cd4374d3e"}
Sep 08 16:46:10.461486 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:10.461444    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p" podStartSLOduration=1.461428834 podStartE2EDuration="1.461428834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:46:10.459611308 +0000 UTC m=+954.622582758" watchObservedRunningTime="2026-09-08 16:46:10.461428834 +0000 UTC m=+954.624400305"
Sep 08 16:46:20.025419 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:20.025380    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:46:20.025864 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:20.025436    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:46:20.038377 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:20.038353    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:46:20.505052 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:20.505025    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:46:22.327584 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:22.327547    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:46:36.630935 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:36.630905    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:46:46.028771 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.028737    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"]
Sep 08 16:46:46.032819 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.032800    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:46:46.036742 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.036517    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-fae677\"/\"explicit-model-fae677-kserve-self-signed-certs\""
Sep 08 16:46:46.036742 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.036564    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-fae677\"/\"kube-root-ca.crt\""
Sep 08 16:46:46.037110 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.037087    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-fae677\"/\"default-dockercfg-5r4kh\""
Sep 08 16:46:46.037345 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.037192    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-fae677\"/\"openshift-service-ca.crt\""
Sep 08 16:46:46.039357 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.039333    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"]
Sep 08 16:46:46.170751 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.170711    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2m26\" (UniqueName: \"kubernetes.io/projected/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-kube-api-access-q2m26\") pod \"explicit-model-fae677-kserve-74c5db6b49-5z6jb\" (UID: \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\") " pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:46:46.170946 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.170777    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-tmp-dir\") pod \"explicit-model-fae677-kserve-74c5db6b49-5z6jb\" (UID: \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\") " pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:46:46.170946 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.170810    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-tls-certs\") pod \"explicit-model-fae677-kserve-74c5db6b49-5z6jb\" (UID: \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\") " pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:46:46.171037 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.170960    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-dshm\") pod \"explicit-model-fae677-kserve-74c5db6b49-5z6jb\" (UID: \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\") " pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:46:46.171037 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.171015    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-home\") pod \"explicit-model-fae677-kserve-74c5db6b49-5z6jb\" (UID: \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\") " pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:46:46.171159 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.171048    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-model-cache\") pod \"explicit-model-fae677-kserve-74c5db6b49-5z6jb\" (UID: \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\") " pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:46:46.272350 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.272317    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-tls-certs\") pod \"explicit-model-fae677-kserve-74c5db6b49-5z6jb\" (UID: \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\") " pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:46:46.272512 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.272386    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-dshm\") pod \"explicit-model-fae677-kserve-74c5db6b49-5z6jb\" (UID: \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\") " pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:46:46.272512 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.272423    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-home\") pod \"explicit-model-fae677-kserve-74c5db6b49-5z6jb\" (UID: \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\") " pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:46:46.272512 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.272451    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-model-cache\") pod \"explicit-model-fae677-kserve-74c5db6b49-5z6jb\" (UID: \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\") " pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:46:46.272512 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.272475    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2m26\" (UniqueName: \"kubernetes.io/projected/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-kube-api-access-q2m26\") pod \"explicit-model-fae677-kserve-74c5db6b49-5z6jb\" (UID: \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\") " pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:46:46.272738 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.272525    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-tmp-dir\") pod \"explicit-model-fae677-kserve-74c5db6b49-5z6jb\" (UID: \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\") " pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:46:46.273117 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.272865    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-tmp-dir\") pod \"explicit-model-fae677-kserve-74c5db6b49-5z6jb\" (UID: \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\") " pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:46:46.273117 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.273029    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-model-cache\") pod \"explicit-model-fae677-kserve-74c5db6b49-5z6jb\" (UID: \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\") " pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:46:46.273117 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.273110    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-home\") pod \"explicit-model-fae677-kserve-74c5db6b49-5z6jb\" (UID: \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\") " pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:46:46.275396 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.275370    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-dshm\") pod \"explicit-model-fae677-kserve-74c5db6b49-5z6jb\" (UID: \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\") " pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:46:46.275748 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.275727    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-tls-certs\") pod \"explicit-model-fae677-kserve-74c5db6b49-5z6jb\" (UID: \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\") " pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:46:46.281626 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.281572    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2m26\" (UniqueName: \"kubernetes.io/projected/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-kube-api-access-q2m26\") pod \"explicit-model-fae677-kserve-74c5db6b49-5z6jb\" (UID: \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\") " pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:46:46.343428 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.343403    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:46:46.616084 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.616059    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"]
Sep 08 16:46:46.617494 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:46:46.617464    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88e537c6_cd10_4f6d_93ef_16cf8c26a77a.slice/crio-e87b919d83438516260c1e83ddb65aa96043d3b65f4f81bf8f398d313488b16e WatchSource:0}: Error finding container e87b919d83438516260c1e83ddb65aa96043d3b65f4f81bf8f398d313488b16e: Status 404 returned error can't find the container with id e87b919d83438516260c1e83ddb65aa96043d3b65f4f81bf8f398d313488b16e
Sep 08 16:46:46.629411 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:46.629388    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb" event={"ID":"88e537c6-cd10-4f6d-93ef-16cf8c26a77a","Type":"ContainerStarted","Data":"e87b919d83438516260c1e83ddb65aa96043d3b65f4f81bf8f398d313488b16e"}
Sep 08 16:46:47.327442 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:47.327412    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:46:47.635771 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:47.635694    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb" event={"ID":"88e537c6-cd10-4f6d-93ef-16cf8c26a77a","Type":"ContainerStarted","Data":"cbadfcf32ddac077d993716e8914e48172a186ca666c10e46f3ba358d6a7fc53"}
Sep 08 16:46:47.652463 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:47.652412    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb" podStartSLOduration=1.6523973760000001 podStartE2EDuration="1.652397376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:46:47.650982521 +0000 UTC m=+991.813953967" watchObservedRunningTime="2026-09-08 16:46:47.652397376 +0000 UTC m=+991.815368855"
Sep 08 16:46:56.347255 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:56.347227    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:46:56.347609 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:56.347267    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:46:56.356486 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:56.356462    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:46:56.694795 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:56.694769    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:46:58.729353 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:46:58.729320    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:47:02.632047 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:02.631988    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"]
Sep 08 16:47:02.757597 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:02.757557    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fae677-d97qt"]
Sep 08 16:47:02.761231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:02.761210    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fae677-d97qt"
Sep 08 16:47:02.763768 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:02.763746    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:47:02.763909 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:02.763750    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-886jf\""
Sep 08 16:47:02.769405 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:02.769279    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fae677-d97qt"]
Sep 08 16:47:02.800172 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:02.800139    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c2bbdb-388d-461b-b213-0e1b8fa3e7b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-fae677-d97qt\" (UID: \"c1c2bbdb-388d-461b-b213-0e1b8fa3e7b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fae677-d97qt"
Sep 08 16:47:02.900808 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:02.900729    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c2bbdb-388d-461b-b213-0e1b8fa3e7b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-fae677-d97qt\" (UID: \"c1c2bbdb-388d-461b-b213-0e1b8fa3e7b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fae677-d97qt"
Sep 08 16:47:02.901382 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:02.901358    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c2bbdb-388d-461b-b213-0e1b8fa3e7b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-fae677-d97qt\" (UID: \"c1c2bbdb-388d-461b-b213-0e1b8fa3e7b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fae677-d97qt"
Sep 08 16:47:03.074418 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:03.074375    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fae677-d97qt"
Sep 08 16:47:03.352049 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:03.352027    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fae677-d97qt"]
Sep 08 16:47:03.353765 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:47:03.353736    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c2bbdb_388d_461b_b213_0e1b8fa3e7b0.slice/crio-68b4f3868cbebba300ba28606c38fd8289cf67afc1abf16c5107d2af2348980c WatchSource:0}: Error finding container 68b4f3868cbebba300ba28606c38fd8289cf67afc1abf16c5107d2af2348980c: Status 404 returned error can't find the container with id 68b4f3868cbebba300ba28606c38fd8289cf67afc1abf16c5107d2af2348980c
Sep 08 16:47:03.720332 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:03.720304    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fae677-d97qt" event={"ID":"c1c2bbdb-388d-461b-b213-0e1b8fa3e7b0","Type":"ContainerStarted","Data":"c63ba6bcc0d3191c30c1455bd372adeb43dc115fd8e7c883e841a80482f421f3"}
Sep 08 16:47:03.720655 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:03.720356    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fae677-d97qt" event={"ID":"c1c2bbdb-388d-461b-b213-0e1b8fa3e7b0","Type":"ContainerStarted","Data":"68b4f3868cbebba300ba28606c38fd8289cf67afc1abf16c5107d2af2348980c"}
Sep 08 16:47:03.734917 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:03.734874    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fae677-d97qt" podStartSLOduration=1.73485984 podStartE2EDuration="1.73485984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:47:03.733490046 +0000 UTC m=+1007.896461494" watchObservedRunningTime="2026-09-08 16:47:03.73485984 +0000 UTC m=+1007.897831288"
Sep 08 16:47:04.727183 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:04.727149    2557 generic.go:356] "Generic (PLEG): container finished" podID="c1c2bbdb-388d-461b-b213-0e1b8fa3e7b0" containerID="c63ba6bcc0d3191c30c1455bd372adeb43dc115fd8e7c883e841a80482f421f3" exitCode=0
Sep 08 16:47:04.727546 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:04.727241    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fae677-d97qt" event={"ID":"c1c2bbdb-388d-461b-b213-0e1b8fa3e7b0","Type":"ContainerDied","Data":"c63ba6bcc0d3191c30c1455bd372adeb43dc115fd8e7c883e841a80482f421f3"}
Sep 08 16:47:05.871044 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:05.871022    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fae677-d97qt"
Sep 08 16:47:05.928112 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:05.928088    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c2bbdb-388d-461b-b213-0e1b8fa3e7b0-maas-api-service-ca\") pod \"c1c2bbdb-388d-461b-b213-0e1b8fa3e7b0\" (UID: \"c1c2bbdb-388d-461b-b213-0e1b8fa3e7b0\") "
Sep 08 16:47:05.928395 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:05.928375    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c2bbdb-388d-461b-b213-0e1b8fa3e7b0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c1c2bbdb-388d-461b-b213-0e1b8fa3e7b0" (UID: "c1c2bbdb-388d-461b-b213-0e1b8fa3e7b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:47:06.028950 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:06.028881    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c2bbdb-388d-461b-b213-0e1b8fa3e7b0-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:47:06.738865 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:06.738814    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fae677-d97qt" event={"ID":"c1c2bbdb-388d-461b-b213-0e1b8fa3e7b0","Type":"ContainerDied","Data":"68b4f3868cbebba300ba28606c38fd8289cf67afc1abf16c5107d2af2348980c"}
Sep 08 16:47:06.738865 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:06.738844    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fae677-d97qt"
Sep 08 16:47:06.739078 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:06.738879    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68b4f3868cbebba300ba28606c38fd8289cf67afc1abf16c5107d2af2348980c"
Sep 08 16:47:09.326726 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:09.326691    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:47:13.375554 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.375515    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq"]
Sep 08 16:47:13.376039 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.375785    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq" podUID="5e45b54e-f8e5-45ad-889e-2b7de92c9f0d" containerName="payload-processing" containerID="cri-o://24e6d57ebc7504eecd75223a94f608198eb008ce9119d53f20b8b6a71f0f8ed2" gracePeriod=30
Sep 08 16:47:13.384366 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.384343    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn"]
Sep 08 16:47:13.384570 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.384535    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn" podUID="d8aaccba-6909-4458-88f2-768bfc829d9b" containerName="payload-pre-processing" containerID="cri-o://045f7d922c9b25b952e2ecdbd753b712f437e22fbd26312308c30230164474ed" gracePeriod=30
Sep 08 16:47:13.554689 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.554660    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fae677-d97qt"]
Sep 08 16:47:13.557479 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.557454    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-fae677-d97qt"]
Sep 08 16:47:13.661671 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.661648    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq"
Sep 08 16:47:13.664374 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.664355    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn"
Sep 08 16:47:13.696985 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.696961    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e45b54e-f8e5-45ad-889e-2b7de92c9f0d-plugins-config-volume\") pod \"5e45b54e-f8e5-45ad-889e-2b7de92c9f0d\" (UID: \"5e45b54e-f8e5-45ad-889e-2b7de92c9f0d\") "
Sep 08 16:47:13.697145 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.696998    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8aaccba-6909-4458-88f2-768bfc829d9b-plugins-config-volume\") pod \"d8aaccba-6909-4458-88f2-768bfc829d9b\" (UID: \"d8aaccba-6909-4458-88f2-768bfc829d9b\") "
Sep 08 16:47:13.697145 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.697027    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97br9\" (UniqueName: \"kubernetes.io/projected/d8aaccba-6909-4458-88f2-768bfc829d9b-kube-api-access-97br9\") pod \"d8aaccba-6909-4458-88f2-768bfc829d9b\" (UID: \"d8aaccba-6909-4458-88f2-768bfc829d9b\") "
Sep 08 16:47:13.697145 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.697045    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcm8w\" (UniqueName: \"kubernetes.io/projected/5e45b54e-f8e5-45ad-889e-2b7de92c9f0d-kube-api-access-kcm8w\") pod \"5e45b54e-f8e5-45ad-889e-2b7de92c9f0d\" (UID: \"5e45b54e-f8e5-45ad-889e-2b7de92c9f0d\") "
Sep 08 16:47:13.697357 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.697333    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8aaccba-6909-4458-88f2-768bfc829d9b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d8aaccba-6909-4458-88f2-768bfc829d9b" (UID: "d8aaccba-6909-4458-88f2-768bfc829d9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:47:13.697424 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.697394    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e45b54e-f8e5-45ad-889e-2b7de92c9f0d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5e45b54e-f8e5-45ad-889e-2b7de92c9f0d" (UID: "5e45b54e-f8e5-45ad-889e-2b7de92c9f0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:47:13.699205 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.699184    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8aaccba-6909-4458-88f2-768bfc829d9b-kube-api-access-97br9" (OuterVolumeSpecName: "kube-api-access-97br9") pod "d8aaccba-6909-4458-88f2-768bfc829d9b" (UID: "d8aaccba-6909-4458-88f2-768bfc829d9b"). InnerVolumeSpecName "kube-api-access-97br9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:47:13.699319 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.699225    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e45b54e-f8e5-45ad-889e-2b7de92c9f0d-kube-api-access-kcm8w" (OuterVolumeSpecName: "kube-api-access-kcm8w") pod "5e45b54e-f8e5-45ad-889e-2b7de92c9f0d" (UID: "5e45b54e-f8e5-45ad-889e-2b7de92c9f0d"). InnerVolumeSpecName "kube-api-access-kcm8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:47:13.775413 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.775384    2557 generic.go:356] "Generic (PLEG): container finished" podID="5e45b54e-f8e5-45ad-889e-2b7de92c9f0d" containerID="24e6d57ebc7504eecd75223a94f608198eb008ce9119d53f20b8b6a71f0f8ed2" exitCode=0
Sep 08 16:47:13.775555 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.775432    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq"
Sep 08 16:47:13.775555 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.775472    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq" event={"ID":"5e45b54e-f8e5-45ad-889e-2b7de92c9f0d","Type":"ContainerDied","Data":"24e6d57ebc7504eecd75223a94f608198eb008ce9119d53f20b8b6a71f0f8ed2"}
Sep 08 16:47:13.775555 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.775528    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq" event={"ID":"5e45b54e-f8e5-45ad-889e-2b7de92c9f0d","Type":"ContainerDied","Data":"0eb499b06b7c4798b16c069745e9f9b7f641223030c385e41554b9b8959f617e"}
Sep 08 16:47:13.775712 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.775561    2557 scope.go:117] "RemoveContainer" containerID="24e6d57ebc7504eecd75223a94f608198eb008ce9119d53f20b8b6a71f0f8ed2"
Sep 08 16:47:13.777031 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.777007    2557 generic.go:356] "Generic (PLEG): container finished" podID="d8aaccba-6909-4458-88f2-768bfc829d9b" containerID="045f7d922c9b25b952e2ecdbd753b712f437e22fbd26312308c30230164474ed" exitCode=0
Sep 08 16:47:13.777207 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.777054    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn"
Sep 08 16:47:13.777207 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.777101    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn" event={"ID":"d8aaccba-6909-4458-88f2-768bfc829d9b","Type":"ContainerDied","Data":"045f7d922c9b25b952e2ecdbd753b712f437e22fbd26312308c30230164474ed"}
Sep 08 16:47:13.777207 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.777166    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn" event={"ID":"d8aaccba-6909-4458-88f2-768bfc829d9b","Type":"ContainerDied","Data":"d1273758bdfe930b403d748f87353e445676951b2e6cc6f1c8a1e62a0c0e41b1"}
Sep 08 16:47:13.784547 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.784527    2557 scope.go:117] "RemoveContainer" containerID="24e6d57ebc7504eecd75223a94f608198eb008ce9119d53f20b8b6a71f0f8ed2"
Sep 08 16:47:13.784890 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:47:13.784869    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e6d57ebc7504eecd75223a94f608198eb008ce9119d53f20b8b6a71f0f8ed2\": container with ID starting with 24e6d57ebc7504eecd75223a94f608198eb008ce9119d53f20b8b6a71f0f8ed2 not found: ID does not exist" containerID="24e6d57ebc7504eecd75223a94f608198eb008ce9119d53f20b8b6a71f0f8ed2"
Sep 08 16:47:13.784956 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.784898    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e6d57ebc7504eecd75223a94f608198eb008ce9119d53f20b8b6a71f0f8ed2"} err="failed to get container status \"24e6d57ebc7504eecd75223a94f608198eb008ce9119d53f20b8b6a71f0f8ed2\": rpc error: code = NotFound desc = could not find container \"24e6d57ebc7504eecd75223a94f608198eb008ce9119d53f20b8b6a71f0f8ed2\": container with ID starting with 24e6d57ebc7504eecd75223a94f608198eb008ce9119d53f20b8b6a71f0f8ed2 not found: ID does not exist"
Sep 08 16:47:13.784956 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.784919    2557 scope.go:117] "RemoveContainer" containerID="045f7d922c9b25b952e2ecdbd753b712f437e22fbd26312308c30230164474ed"
Sep 08 16:47:13.793111 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.793094    2557 scope.go:117] "RemoveContainer" containerID="045f7d922c9b25b952e2ecdbd753b712f437e22fbd26312308c30230164474ed"
Sep 08 16:47:13.793365 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:47:13.793349    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"045f7d922c9b25b952e2ecdbd753b712f437e22fbd26312308c30230164474ed\": container with ID starting with 045f7d922c9b25b952e2ecdbd753b712f437e22fbd26312308c30230164474ed not found: ID does not exist" containerID="045f7d922c9b25b952e2ecdbd753b712f437e22fbd26312308c30230164474ed"
Sep 08 16:47:13.793410 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.793371    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"045f7d922c9b25b952e2ecdbd753b712f437e22fbd26312308c30230164474ed"} err="failed to get container status \"045f7d922c9b25b952e2ecdbd753b712f437e22fbd26312308c30230164474ed\": rpc error: code = NotFound desc = could not find container \"045f7d922c9b25b952e2ecdbd753b712f437e22fbd26312308c30230164474ed\": container with ID starting with 045f7d922c9b25b952e2ecdbd753b712f437e22fbd26312308c30230164474ed not found: ID does not exist"
Sep 08 16:47:13.797661 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.797643    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e45b54e-f8e5-45ad-889e-2b7de92c9f0d-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:47:13.797732 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.797663    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8aaccba-6909-4458-88f2-768bfc829d9b-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:47:13.797732 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.797672    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97br9\" (UniqueName: \"kubernetes.io/projected/d8aaccba-6909-4458-88f2-768bfc829d9b-kube-api-access-97br9\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:47:13.797732 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.797681    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcm8w\" (UniqueName: \"kubernetes.io/projected/5e45b54e-f8e5-45ad-889e-2b7de92c9f0d-kube-api-access-kcm8w\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:47:13.810242 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.810223    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn"]
Sep 08 16:47:13.817385 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.817365    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-fae677-758795bd6b-cwkqn"]
Sep 08 16:47:13.835387 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.835365    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq"]
Sep 08 16:47:13.844109 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:13.844089    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-fae677-7dc8dfb8df-725hq"]
Sep 08 16:47:14.346072 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:14.346042    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e45b54e-f8e5-45ad-889e-2b7de92c9f0d" path="/var/lib/kubelet/pods/5e45b54e-f8e5-45ad-889e-2b7de92c9f0d/volumes"
Sep 08 16:47:14.346353 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:14.346340    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1c2bbdb-388d-461b-b213-0e1b8fa3e7b0" path="/var/lib/kubelet/pods/c1c2bbdb-388d-461b-b213-0e1b8fa3e7b0/volumes"
Sep 08 16:47:14.346647 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:14.346635    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8aaccba-6909-4458-88f2-768bfc829d9b" path="/var/lib/kubelet/pods/d8aaccba-6909-4458-88f2-768bfc829d9b/volumes"
Sep 08 16:47:17.673744 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:17.673703    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb" podUID="88e537c6-cd10-4f6d-93ef-16cf8c26a77a" containerName="main" containerID="cri-o://cbadfcf32ddac077d993716e8914e48172a186ca666c10e46f3ba358d6a7fc53" gracePeriod=45
Sep 08 16:47:17.801194 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:17.801162    2557 generic.go:356] "Generic (PLEG): container finished" podID="88e537c6-cd10-4f6d-93ef-16cf8c26a77a" containerID="cbadfcf32ddac077d993716e8914e48172a186ca666c10e46f3ba358d6a7fc53" exitCode=0
Sep 08 16:47:17.801286 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:17.801224    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb" event={"ID":"88e537c6-cd10-4f6d-93ef-16cf8c26a77a","Type":"ContainerDied","Data":"cbadfcf32ddac077d993716e8914e48172a186ca666c10e46f3ba358d6a7fc53"}
Sep 08 16:47:17.920467 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:17.920441    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:47:18.033512 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:18.033419    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-tls-certs\") pod \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\" (UID: \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\") "
Sep 08 16:47:18.033512 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:18.033456    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-dshm\") pod \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\" (UID: \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\") "
Sep 08 16:47:18.033512 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:18.033476    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2m26\" (UniqueName: \"kubernetes.io/projected/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-kube-api-access-q2m26\") pod \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\" (UID: \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\") "
Sep 08 16:47:18.033797 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:18.033602    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-tmp-dir\") pod \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\" (UID: \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\") "
Sep 08 16:47:18.033797 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:18.033631    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-model-cache\") pod \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\" (UID: \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\") "
Sep 08 16:47:18.033797 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:18.033713    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-home\") pod \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\" (UID: \"88e537c6-cd10-4f6d-93ef-16cf8c26a77a\") "
Sep 08 16:47:18.033997 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:18.033898    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88e537c6-cd10-4f6d-93ef-16cf8c26a77a" (UID: "88e537c6-cd10-4f6d-93ef-16cf8c26a77a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:47:18.033997 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:18.033928    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-model-cache" (OuterVolumeSpecName: "model-cache") pod "88e537c6-cd10-4f6d-93ef-16cf8c26a77a" (UID: "88e537c6-cd10-4f6d-93ef-16cf8c26a77a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:47:18.034098 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:18.034012    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-home" (OuterVolumeSpecName: "home") pod "88e537c6-cd10-4f6d-93ef-16cf8c26a77a" (UID: "88e537c6-cd10-4f6d-93ef-16cf8c26a77a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:47:18.034098 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:18.034075    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:47:18.034098 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:18.034089    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:47:18.035910 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:18.035890    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-dshm" (OuterVolumeSpecName: "dshm") pod "88e537c6-cd10-4f6d-93ef-16cf8c26a77a" (UID: "88e537c6-cd10-4f6d-93ef-16cf8c26a77a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:47:18.035910 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:18.035901    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88e537c6-cd10-4f6d-93ef-16cf8c26a77a" (UID: "88e537c6-cd10-4f6d-93ef-16cf8c26a77a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:47:18.036042 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:18.035940    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-kube-api-access-q2m26" (OuterVolumeSpecName: "kube-api-access-q2m26") pod "88e537c6-cd10-4f6d-93ef-16cf8c26a77a" (UID: "88e537c6-cd10-4f6d-93ef-16cf8c26a77a"). InnerVolumeSpecName "kube-api-access-q2m26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:47:18.134588 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:18.134559    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:47:18.134588 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:18.134584    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:47:18.134588 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:18.134594    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:47:18.134787 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:18.134602    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2m26\" (UniqueName: \"kubernetes.io/projected/88e537c6-cd10-4f6d-93ef-16cf8c26a77a-kube-api-access-q2m26\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:47:18.807266 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:18.807237    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"
Sep 08 16:47:18.807266 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:18.807241    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb" event={"ID":"88e537c6-cd10-4f6d-93ef-16cf8c26a77a","Type":"ContainerDied","Data":"e87b919d83438516260c1e83ddb65aa96043d3b65f4f81bf8f398d313488b16e"}
Sep 08 16:47:18.807773 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:18.807295    2557 scope.go:117] "RemoveContainer" containerID="cbadfcf32ddac077d993716e8914e48172a186ca666c10e46f3ba358d6a7fc53"
Sep 08 16:47:18.825323 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:18.825299    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"]
Sep 08 16:47:18.829014 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:18.828991    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-fae677/explicit-model-fae677-kserve-74c5db6b49-5z6jb"]
Sep 08 16:47:20.243201 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:20.243166    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:47:20.345368 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:20.345333    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88e537c6-cd10-4f6d-93ef-16cf8c26a77a" path="/var/lib/kubelet/pods/88e537c6-cd10-4f6d-93ef-16cf8c26a77a/volumes"
Sep 08 16:47:43.795694 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:43.795663    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx"]
Sep 08 16:47:43.796169 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:43.796031    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1c2bbdb-388d-461b-b213-0e1b8fa3e7b0" containerName="revoke-keys"
Sep 08 16:47:43.796169 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:43.796042    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1c2bbdb-388d-461b-b213-0e1b8fa3e7b0" containerName="revoke-keys"
Sep 08 16:47:43.796169 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:43.796051    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8aaccba-6909-4458-88f2-768bfc829d9b" containerName="payload-pre-processing"
Sep 08 16:47:43.796169 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:43.796057    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8aaccba-6909-4458-88f2-768bfc829d9b" containerName="payload-pre-processing"
Sep 08 16:47:43.796169 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:43.796067    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e537c6-cd10-4f6d-93ef-16cf8c26a77a" containerName="main"
Sep 08 16:47:43.796169 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:43.796073    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e537c6-cd10-4f6d-93ef-16cf8c26a77a" containerName="main"
Sep 08 16:47:43.796169 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:43.796088    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e45b54e-f8e5-45ad-889e-2b7de92c9f0d" containerName="payload-processing"
Sep 08 16:47:43.796169 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:43.796093    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e45b54e-f8e5-45ad-889e-2b7de92c9f0d" containerName="payload-processing"
Sep 08 16:47:43.796169 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:43.796144    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e45b54e-f8e5-45ad-889e-2b7de92c9f0d" containerName="payload-processing"
Sep 08 16:47:43.796169 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:43.796149    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8aaccba-6909-4458-88f2-768bfc829d9b" containerName="payload-pre-processing"
Sep 08 16:47:43.796169 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:43.796155    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="88e537c6-cd10-4f6d-93ef-16cf8c26a77a" containerName="main"
Sep 08 16:47:43.796169 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:43.796162    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1c2bbdb-388d-461b-b213-0e1b8fa3e7b0" containerName="revoke-keys"
Sep 08 16:47:43.798966 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:43.798950    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx"
Sep 08 16:47:43.801472 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:43.801449    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-bdae4bb7-dockercfg-z4cr9\""
Sep 08 16:47:43.801701 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:43.801687    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-bdae4bb7\""
Sep 08 16:47:43.811827 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:43.811807    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx"]
Sep 08 16:47:43.821232 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:43.821212    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm"]
Sep 08 16:47:43.824289 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:43.824275    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm"
Sep 08 16:47:43.833333 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:43.833314    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm"]
Sep 08 16:47:43.938991 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:43.938962    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f1ca04-44b4-48b1-9fb7-3da2f8c29814-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx\" (UID: \"c3f1ca04-44b4-48b1-9fb7-3da2f8c29814\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx"
Sep 08 16:47:43.939141 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:43.939019    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgcz4\" (UniqueName: \"kubernetes.io/projected/c3f1ca04-44b4-48b1-9fb7-3da2f8c29814-kube-api-access-xgcz4\") pod \"payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx\" (UID: \"c3f1ca04-44b4-48b1-9fb7-3da2f8c29814\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx"
Sep 08 16:47:43.939141 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:43.939124    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-584bh\" (UniqueName: \"kubernetes.io/projected/70bf6456-34db-4039-8d3c-21b1355a21c2-kube-api-access-584bh\") pod \"payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm\" (UID: \"70bf6456-34db-4039-8d3c-21b1355a21c2\") " pod="openshift-ingress/payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm"
Sep 08 16:47:43.939223 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:43.939149    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70bf6456-34db-4039-8d3c-21b1355a21c2-plugins-config-volume\") pod \"payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm\" (UID: \"70bf6456-34db-4039-8d3c-21b1355a21c2\") " pod="openshift-ingress/payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm"
Sep 08 16:47:44.039748 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:44.039713    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f1ca04-44b4-48b1-9fb7-3da2f8c29814-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx\" (UID: \"c3f1ca04-44b4-48b1-9fb7-3da2f8c29814\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx"
Sep 08 16:47:44.039957 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:44.039766    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgcz4\" (UniqueName: \"kubernetes.io/projected/c3f1ca04-44b4-48b1-9fb7-3da2f8c29814-kube-api-access-xgcz4\") pod \"payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx\" (UID: \"c3f1ca04-44b4-48b1-9fb7-3da2f8c29814\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx"
Sep 08 16:47:44.039957 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:44.039800    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-584bh\" (UniqueName: \"kubernetes.io/projected/70bf6456-34db-4039-8d3c-21b1355a21c2-kube-api-access-584bh\") pod \"payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm\" (UID: \"70bf6456-34db-4039-8d3c-21b1355a21c2\") " pod="openshift-ingress/payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm"
Sep 08 16:47:44.039957 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:44.039817    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70bf6456-34db-4039-8d3c-21b1355a21c2-plugins-config-volume\") pod \"payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm\" (UID: \"70bf6456-34db-4039-8d3c-21b1355a21c2\") " pod="openshift-ingress/payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm"
Sep 08 16:47:44.040564 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:44.040544    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70bf6456-34db-4039-8d3c-21b1355a21c2-plugins-config-volume\") pod \"payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm\" (UID: \"70bf6456-34db-4039-8d3c-21b1355a21c2\") " pod="openshift-ingress/payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm"
Sep 08 16:47:44.040651 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:44.040633    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f1ca04-44b4-48b1-9fb7-3da2f8c29814-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx\" (UID: \"c3f1ca04-44b4-48b1-9fb7-3da2f8c29814\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx"
Sep 08 16:47:44.047812 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:44.047752    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-584bh\" (UniqueName: \"kubernetes.io/projected/70bf6456-34db-4039-8d3c-21b1355a21c2-kube-api-access-584bh\") pod \"payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm\" (UID: \"70bf6456-34db-4039-8d3c-21b1355a21c2\") " pod="openshift-ingress/payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm"
Sep 08 16:47:44.047930 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:44.047857    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgcz4\" (UniqueName: \"kubernetes.io/projected/c3f1ca04-44b4-48b1-9fb7-3da2f8c29814-kube-api-access-xgcz4\") pod \"payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx\" (UID: \"c3f1ca04-44b4-48b1-9fb7-3da2f8c29814\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx"
Sep 08 16:47:44.109918 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:44.109890    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx"
Sep 08 16:47:44.135125 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:44.135100    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm"
Sep 08 16:47:44.381787 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:44.381708    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx"]
Sep 08 16:47:44.383910 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:47:44.383881    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f1ca04_44b4_48b1_9fb7_3da2f8c29814.slice/crio-dc67dca8f095b7baba07b120f353ea4123751a6f8d910ca39a9f198b4907996f WatchSource:0}: Error finding container dc67dca8f095b7baba07b120f353ea4123751a6f8d910ca39a9f198b4907996f: Status 404 returned error can't find the container with id dc67dca8f095b7baba07b120f353ea4123751a6f8d910ca39a9f198b4907996f
Sep 08 16:47:44.417620 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:44.417600    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm"]
Sep 08 16:47:44.422815 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:47:44.422794    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70bf6456_34db_4039_8d3c_21b1355a21c2.slice/crio-56b62228e79c1e81582f36145ced82872f49d621a25d294604a4de5d0bae5f50 WatchSource:0}: Error finding container 56b62228e79c1e81582f36145ced82872f49d621a25d294604a4de5d0bae5f50: Status 404 returned error can't find the container with id 56b62228e79c1e81582f36145ced82872f49d621a25d294604a4de5d0bae5f50
Sep 08 16:47:44.954419 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:44.954379    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx" event={"ID":"c3f1ca04-44b4-48b1-9fb7-3da2f8c29814","Type":"ContainerStarted","Data":"19c9677d3adf53d834ddbf61cdc4c685ebbbb1d61a5c71a1dcdae36933027147"}
Sep 08 16:47:44.954886 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:44.954539    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx"
Sep 08 16:47:44.954886 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:44.954572    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx" event={"ID":"c3f1ca04-44b4-48b1-9fb7-3da2f8c29814","Type":"ContainerStarted","Data":"dc67dca8f095b7baba07b120f353ea4123751a6f8d910ca39a9f198b4907996f"}
Sep 08 16:47:44.955988 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:44.955960    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm" event={"ID":"70bf6456-34db-4039-8d3c-21b1355a21c2","Type":"ContainerStarted","Data":"f186a539a8e9cb366998d74d22bfbe7acff9af5a80a7ea09970ae94e2c5ba6b8"}
Sep 08 16:47:44.956129 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:44.956006    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm" event={"ID":"70bf6456-34db-4039-8d3c-21b1355a21c2","Type":"ContainerStarted","Data":"56b62228e79c1e81582f36145ced82872f49d621a25d294604a4de5d0bae5f50"}
Sep 08 16:47:44.956129 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:44.956063    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm"
Sep 08 16:47:44.991733 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:44.991684    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm" podStartSLOduration=1.9916674479999998 podStartE2EDuration="1.991667448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:47:44.990583924 +0000 UTC m=+1049.153555366" watchObservedRunningTime="2026-09-08 16:47:44.991667448 +0000 UTC m=+1049.154638901"
Sep 08 16:47:44.992661 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:44.992629    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx" podStartSLOduration=1.992614846 podStartE2EDuration="1.992614846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:47:44.972890552 +0000 UTC m=+1049.135862002" watchObservedRunningTime="2026-09-08 16:47:44.992614846 +0000 UTC m=+1049.155586296"
Sep 08 16:47:50.615987 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.615953    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"]
Sep 08 16:47:50.619714 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.619691    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:47:50.623753 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.623732    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-51b8ca\"/\"noten-model-51b8ca-kserve-self-signed-certs\""
Sep 08 16:47:50.623864 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.623731    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-51b8ca\"/\"kube-root-ca.crt\""
Sep 08 16:47:50.623864 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.623741    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-51b8ca\"/\"default-dockercfg-vpqmm\""
Sep 08 16:47:50.623994 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.623806    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-51b8ca\"/\"openshift-service-ca.crt\""
Sep 08 16:47:50.629621 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.629601    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"]
Sep 08 16:47:50.798778 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.798744    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-tls-certs\") pod \"noten-model-51b8ca-kserve-6676799c8c-99j6k\" (UID: \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\") " pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:47:50.798967 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.798785    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-home\") pod \"noten-model-51b8ca-kserve-6676799c8c-99j6k\" (UID: \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\") " pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:47:50.798967 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.798819    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-model-cache\") pod \"noten-model-51b8ca-kserve-6676799c8c-99j6k\" (UID: \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\") " pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:47:50.798967 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.798905    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-tmp-dir\") pod \"noten-model-51b8ca-kserve-6676799c8c-99j6k\" (UID: \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\") " pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:47:50.798967 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.798944    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wghjm\" (UniqueName: \"kubernetes.io/projected/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-kube-api-access-wghjm\") pod \"noten-model-51b8ca-kserve-6676799c8c-99j6k\" (UID: \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\") " pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:47:50.799124 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.798977    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-dshm\") pod \"noten-model-51b8ca-kserve-6676799c8c-99j6k\" (UID: \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\") " pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:47:50.900217 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.900119    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-dshm\") pod \"noten-model-51b8ca-kserve-6676799c8c-99j6k\" (UID: \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\") " pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:47:50.900217 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.900172    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-tls-certs\") pod \"noten-model-51b8ca-kserve-6676799c8c-99j6k\" (UID: \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\") " pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:47:50.900217 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.900193    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-home\") pod \"noten-model-51b8ca-kserve-6676799c8c-99j6k\" (UID: \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\") " pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:47:50.900495 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.900222    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-model-cache\") pod \"noten-model-51b8ca-kserve-6676799c8c-99j6k\" (UID: \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\") " pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:47:50.900495 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.900266    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-tmp-dir\") pod \"noten-model-51b8ca-kserve-6676799c8c-99j6k\" (UID: \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\") " pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:47:50.900495 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.900304    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wghjm\" (UniqueName: \"kubernetes.io/projected/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-kube-api-access-wghjm\") pod \"noten-model-51b8ca-kserve-6676799c8c-99j6k\" (UID: \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\") " pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:47:50.900663 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.900605    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-home\") pod \"noten-model-51b8ca-kserve-6676799c8c-99j6k\" (UID: \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\") " pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:47:50.900953 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.900887    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-tmp-dir\") pod \"noten-model-51b8ca-kserve-6676799c8c-99j6k\" (UID: \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\") " pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:47:50.901065 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.901007    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-model-cache\") pod \"noten-model-51b8ca-kserve-6676799c8c-99j6k\" (UID: \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\") " pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:47:50.902822 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.902796    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-dshm\") pod \"noten-model-51b8ca-kserve-6676799c8c-99j6k\" (UID: \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\") " pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:47:50.903377 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.903354    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-tls-certs\") pod \"noten-model-51b8ca-kserve-6676799c8c-99j6k\" (UID: \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\") " pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:47:50.908986 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.908958    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wghjm\" (UniqueName: \"kubernetes.io/projected/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-kube-api-access-wghjm\") pod \"noten-model-51b8ca-kserve-6676799c8c-99j6k\" (UID: \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\") " pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:47:50.930773 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:50.930746    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:47:51.205737 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:51.205600    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"]
Sep 08 16:47:51.208035 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:47:51.208004    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6d2986e_6c33_426c_bd41_eee81cdf9cfa.slice/crio-bfd16f3e377a0ca1a93cd47d60f7d4bb3f6f7d6108a02be0478867e524311d0d WatchSource:0}: Error finding container bfd16f3e377a0ca1a93cd47d60f7d4bb3f6f7d6108a02be0478867e524311d0d: Status 404 returned error can't find the container with id bfd16f3e377a0ca1a93cd47d60f7d4bb3f6f7d6108a02be0478867e524311d0d
Sep 08 16:47:51.234605 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:51.234582    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:47:52.000217 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:52.000179    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k" event={"ID":"b6d2986e-6c33-426c-bd41-eee81cdf9cfa","Type":"ContainerStarted","Data":"b23828661fe4dbf8d8fa732080a246d198e193951c3b09a99bb3e821ba26eab3"}
Sep 08 16:47:52.000618 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:52.000229    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k" event={"ID":"b6d2986e-6c33-426c-bd41-eee81cdf9cfa","Type":"ContainerStarted","Data":"bfd16f3e377a0ca1a93cd47d60f7d4bb3f6f7d6108a02be0478867e524311d0d"}
Sep 08 16:47:55.964278 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:55.964247    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx"
Sep 08 16:47:55.964697 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:55.964316    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm"
Sep 08 16:47:55.980753 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:47:55.980704    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k" podStartSLOduration=5.980686461 podStartE2EDuration="5.980686461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:47:52.016357305 +0000 UTC m=+1056.179328754" watchObservedRunningTime="2026-09-08 16:47:55.980686461 +0000 UTC m=+1060.143657906"
Sep 08 16:48:00.931433 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:00.931396    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:48:00.931801 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:00.931444    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:48:00.944050 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:00.944026    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:48:01.058066 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:01.058041    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:48:02.425317 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:02.425280    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:48:07.008010 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:07.007863    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"]
Sep 08 16:48:08.115693 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:08.115658    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"]
Sep 08 16:48:08.227511 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:08.227474    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c9b098-ww6gd"]
Sep 08 16:48:08.231013 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:08.230997    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c9b098-ww6gd"
Sep 08 16:48:08.233633 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:08.233609    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:48:08.233633 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:08.233635    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-886jf\""
Sep 08 16:48:08.238985 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:08.238745    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c9b098-ww6gd"]
Sep 08 16:48:08.347439 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:08.347410    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcaefa32-3362-485f-addf-7bff8ed381b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c9b098-ww6gd\" (UID: \"dcaefa32-3362-485f-addf-7bff8ed381b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c9b098-ww6gd"
Sep 08 16:48:08.448569 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:08.448534    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcaefa32-3362-485f-addf-7bff8ed381b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c9b098-ww6gd\" (UID: \"dcaefa32-3362-485f-addf-7bff8ed381b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c9b098-ww6gd"
Sep 08 16:48:08.449215 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:08.449193    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcaefa32-3362-485f-addf-7bff8ed381b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c9b098-ww6gd\" (UID: \"dcaefa32-3362-485f-addf-7bff8ed381b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c9b098-ww6gd"
Sep 08 16:48:08.544006 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:08.543975    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c9b098-ww6gd"
Sep 08 16:48:08.819551 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:08.819523    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c9b098-ww6gd"]
Sep 08 16:48:08.821432 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:48:08.821404    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcaefa32_3362_485f_addf_7bff8ed381b8.slice/crio-c3c6ce1373bb9f380c55f2a55b92446479b586e1c575b57e6af7e123f4ba7f7e WatchSource:0}: Error finding container c3c6ce1373bb9f380c55f2a55b92446479b586e1c575b57e6af7e123f4ba7f7e: Status 404 returned error can't find the container with id c3c6ce1373bb9f380c55f2a55b92446479b586e1c575b57e6af7e123f4ba7f7e
Sep 08 16:48:09.100436 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:09.100399    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c9b098-ww6gd" event={"ID":"dcaefa32-3362-485f-addf-7bff8ed381b8","Type":"ContainerStarted","Data":"c3c6ce1373bb9f380c55f2a55b92446479b586e1c575b57e6af7e123f4ba7f7e"}
Sep 08 16:48:10.106876 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:10.106760    2557 generic.go:356] "Generic (PLEG): container finished" podID="dcaefa32-3362-485f-addf-7bff8ed381b8" containerID="722e48cba9789f9d213ac31f97b7cd338d16f2b08a766e5a3c57ff1c6b560352" exitCode=0
Sep 08 16:48:10.106876 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:10.106795    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c9b098-ww6gd" event={"ID":"dcaefa32-3362-485f-addf-7bff8ed381b8","Type":"ContainerDied","Data":"722e48cba9789f9d213ac31f97b7cd338d16f2b08a766e5a3c57ff1c6b560352"}
Sep 08 16:48:11.250265 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:11.250242    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c9b098-ww6gd"
Sep 08 16:48:11.276731 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:11.276706    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcaefa32-3362-485f-addf-7bff8ed381b8-maas-api-service-ca\") pod \"dcaefa32-3362-485f-addf-7bff8ed381b8\" (UID: \"dcaefa32-3362-485f-addf-7bff8ed381b8\") "
Sep 08 16:48:11.277080 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:11.277056    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcaefa32-3362-485f-addf-7bff8ed381b8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dcaefa32-3362-485f-addf-7bff8ed381b8" (UID: "dcaefa32-3362-485f-addf-7bff8ed381b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:48:11.377727 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:11.377652    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcaefa32-3362-485f-addf-7bff8ed381b8-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:12.118345 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:12.118310    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c9b098-ww6gd"
Sep 08 16:48:12.118535 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:12.118343    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c9b098-ww6gd" event={"ID":"dcaefa32-3362-485f-addf-7bff8ed381b8","Type":"ContainerDied","Data":"c3c6ce1373bb9f380c55f2a55b92446479b586e1c575b57e6af7e123f4ba7f7e"}
Sep 08 16:48:12.118535 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:12.118429    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3c6ce1373bb9f380c55f2a55b92446479b586e1c575b57e6af7e123f4ba7f7e"
Sep 08 16:48:12.735525 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:12.735493    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:48:14.640207 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:14.640173    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdae4bb7-jmzhz"]
Sep 08 16:48:14.640809 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:14.640789    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcaefa32-3362-485f-addf-7bff8ed381b8" containerName="revoke-keys"
Sep 08 16:48:14.640915 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:14.640813    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcaefa32-3362-485f-addf-7bff8ed381b8" containerName="revoke-keys"
Sep 08 16:48:14.640973 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:14.640924    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcaefa32-3362-485f-addf-7bff8ed381b8" containerName="revoke-keys"
Sep 08 16:48:14.644028 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:14.644007    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdae4bb7-jmzhz"
Sep 08 16:48:14.646717 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:14.646700    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-886jf\""
Sep 08 16:48:14.646717 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:14.646710    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:48:14.650153 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:14.650132    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdae4bb7-jmzhz"]
Sep 08 16:48:14.708272 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:14.708243    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15268038-3db7-4ef7-9945-81f80f74d4a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bdae4bb7-jmzhz\" (UID: \"15268038-3db7-4ef7-9945-81f80f74d4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdae4bb7-jmzhz"
Sep 08 16:48:14.809019 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:14.808989    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15268038-3db7-4ef7-9945-81f80f74d4a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bdae4bb7-jmzhz\" (UID: \"15268038-3db7-4ef7-9945-81f80f74d4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdae4bb7-jmzhz"
Sep 08 16:48:14.809615 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:14.809593    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15268038-3db7-4ef7-9945-81f80f74d4a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bdae4bb7-jmzhz\" (UID: \"15268038-3db7-4ef7-9945-81f80f74d4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdae4bb7-jmzhz"
Sep 08 16:48:14.957195 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:14.957164    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdae4bb7-jmzhz"
Sep 08 16:48:15.226875 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:15.226848    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdae4bb7-jmzhz"]
Sep 08 16:48:15.228690 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:48:15.228653    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15268038_3db7_4ef7_9945_81f80f74d4a8.slice/crio-e45f5a9d3ef390bd7a28857b19483a9db141aa8bc7ef8277179bd0df88384e1f WatchSource:0}: Error finding container e45f5a9d3ef390bd7a28857b19483a9db141aa8bc7ef8277179bd0df88384e1f: Status 404 returned error can't find the container with id e45f5a9d3ef390bd7a28857b19483a9db141aa8bc7ef8277179bd0df88384e1f
Sep 08 16:48:16.140114 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:16.140014    2557 generic.go:356] "Generic (PLEG): container finished" podID="15268038-3db7-4ef7-9945-81f80f74d4a8" containerID="4c67d9354f00859d58823966571976366ce61a3d8e9715217edfdd35f78a34aa" exitCode=0
Sep 08 16:48:16.140457 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:16.140109    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdae4bb7-jmzhz" event={"ID":"15268038-3db7-4ef7-9945-81f80f74d4a8","Type":"ContainerDied","Data":"4c67d9354f00859d58823966571976366ce61a3d8e9715217edfdd35f78a34aa"}
Sep 08 16:48:16.140457 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:16.140163    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdae4bb7-jmzhz" event={"ID":"15268038-3db7-4ef7-9945-81f80f74d4a8","Type":"ContainerStarted","Data":"e45f5a9d3ef390bd7a28857b19483a9db141aa8bc7ef8277179bd0df88384e1f"}
Sep 08 16:48:17.277647 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:17.277622    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdae4bb7-jmzhz"
Sep 08 16:48:17.330559 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:17.330523    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15268038-3db7-4ef7-9945-81f80f74d4a8-maas-api-service-ca\") pod \"15268038-3db7-4ef7-9945-81f80f74d4a8\" (UID: \"15268038-3db7-4ef7-9945-81f80f74d4a8\") "
Sep 08 16:48:17.330872 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:17.330849    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15268038-3db7-4ef7-9945-81f80f74d4a8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "15268038-3db7-4ef7-9945-81f80f74d4a8" (UID: "15268038-3db7-4ef7-9945-81f80f74d4a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:48:17.431783 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:17.431752    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15268038-3db7-4ef7-9945-81f80f74d4a8-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:18.151039 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:18.151016    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdae4bb7-jmzhz"
Sep 08 16:48:18.151212 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:18.151015    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdae4bb7-jmzhz" event={"ID":"15268038-3db7-4ef7-9945-81f80f74d4a8","Type":"ContainerDied","Data":"e45f5a9d3ef390bd7a28857b19483a9db141aa8bc7ef8277179bd0df88384e1f"}
Sep 08 16:48:18.151212 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:18.151136    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e45f5a9d3ef390bd7a28857b19483a9db141aa8bc7ef8277179bd0df88384e1f"
Sep 08 16:48:19.010154 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:19.010117    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942"]
Sep 08 16:48:19.010623 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:19.010362    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942" podUID="776168c6-1e80-46df-bf8a-b66a38f678a6" containerName="payload-processing" containerID="cri-o://37dae651d7c0fddf48cb8d142a361ddf510f796626f5a4356c3fab17fffaf25e" gracePeriod=30
Sep 08 16:48:19.018810 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:19.018781    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5"]
Sep 08 16:48:19.019043 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:19.019016    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5" podUID="db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d" containerName="payload-pre-processing" containerID="cri-o://4c637d572cb5236ff12f8900c94ed1bd07f8eec425becf403df2de76177683e0" gracePeriod=30
Sep 08 16:48:19.159808 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:19.159780    2557 generic.go:356] "Generic (PLEG): container finished" podID="db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d" containerID="4c637d572cb5236ff12f8900c94ed1bd07f8eec425becf403df2de76177683e0" exitCode=0
Sep 08 16:48:19.159948 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:19.159901    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5" event={"ID":"db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d","Type":"ContainerDied","Data":"4c637d572cb5236ff12f8900c94ed1bd07f8eec425becf403df2de76177683e0"}
Sep 08 16:48:19.161554 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:19.161535    2557 generic.go:356] "Generic (PLEG): container finished" podID="776168c6-1e80-46df-bf8a-b66a38f678a6" containerID="37dae651d7c0fddf48cb8d142a361ddf510f796626f5a4356c3fab17fffaf25e" exitCode=0
Sep 08 16:48:19.161668 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:19.161571    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942" event={"ID":"776168c6-1e80-46df-bf8a-b66a38f678a6","Type":"ContainerDied","Data":"37dae651d7c0fddf48cb8d142a361ddf510f796626f5a4356c3fab17fffaf25e"}
Sep 08 16:48:19.181087 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:19.181058    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c9b098-ww6gd"]
Sep 08 16:48:19.183979 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:19.183957    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c9b098-ww6gd"]
Sep 08 16:48:19.304054 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:19.304026    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942"
Sep 08 16:48:19.306671 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:19.306653    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5"
Sep 08 16:48:19.447089 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:19.447063    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d-plugins-config-volume\") pod \"db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d\" (UID: \"db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d\") "
Sep 08 16:48:19.447089 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:19.447099    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/776168c6-1e80-46df-bf8a-b66a38f678a6-plugins-config-volume\") pod \"776168c6-1e80-46df-bf8a-b66a38f678a6\" (UID: \"776168c6-1e80-46df-bf8a-b66a38f678a6\") "
Sep 08 16:48:19.447327 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:19.447138    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7zhk\" (UniqueName: \"kubernetes.io/projected/db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d-kube-api-access-h7zhk\") pod \"db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d\" (UID: \"db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d\") "
Sep 08 16:48:19.447327 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:19.447156    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7qv9\" (UniqueName: \"kubernetes.io/projected/776168c6-1e80-46df-bf8a-b66a38f678a6-kube-api-access-f7qv9\") pod \"776168c6-1e80-46df-bf8a-b66a38f678a6\" (UID: \"776168c6-1e80-46df-bf8a-b66a38f678a6\") "
Sep 08 16:48:19.447519 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:19.447496    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/776168c6-1e80-46df-bf8a-b66a38f678a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "776168c6-1e80-46df-bf8a-b66a38f678a6" (UID: "776168c6-1e80-46df-bf8a-b66a38f678a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:48:19.447575 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:19.447496    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d" (UID: "db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:48:19.449505 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:19.449483    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d-kube-api-access-h7zhk" (OuterVolumeSpecName: "kube-api-access-h7zhk") pod "db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d" (UID: "db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d"). InnerVolumeSpecName "kube-api-access-h7zhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:48:19.449584 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:19.449502    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776168c6-1e80-46df-bf8a-b66a38f678a6-kube-api-access-f7qv9" (OuterVolumeSpecName: "kube-api-access-f7qv9") pod "776168c6-1e80-46df-bf8a-b66a38f678a6" (UID: "776168c6-1e80-46df-bf8a-b66a38f678a6"). InnerVolumeSpecName "kube-api-access-f7qv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:48:19.548692 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:19.548635    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:19.548692 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:19.548657    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/776168c6-1e80-46df-bf8a-b66a38f678a6-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:19.548692 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:19.548667    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7zhk\" (UniqueName: \"kubernetes.io/projected/db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d-kube-api-access-h7zhk\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:19.548692 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:19.548675    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7qv9\" (UniqueName: \"kubernetes.io/projected/776168c6-1e80-46df-bf8a-b66a38f678a6-kube-api-access-f7qv9\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:20.167630 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:20.167600    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5"
Sep 08 16:48:20.168119 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:20.167637    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5" event={"ID":"db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d","Type":"ContainerDied","Data":"f65328ad34607857498ec2314b047778bc52eb36839be4e7aa2fdf5b974d4958"}
Sep 08 16:48:20.168119 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:20.167712    2557 scope.go:117] "RemoveContainer" containerID="4c637d572cb5236ff12f8900c94ed1bd07f8eec425becf403df2de76177683e0"
Sep 08 16:48:20.169135 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:20.168968    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942" event={"ID":"776168c6-1e80-46df-bf8a-b66a38f678a6","Type":"ContainerDied","Data":"6a996591e679a7c2f09b92b266fbacb7deb2db9038d4163dc866002bf39079ef"}
Sep 08 16:48:20.169135 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:20.168983    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942"
Sep 08 16:48:20.178292 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:20.178193    2557 scope.go:117] "RemoveContainer" containerID="37dae651d7c0fddf48cb8d142a361ddf510f796626f5a4356c3fab17fffaf25e"
Sep 08 16:48:20.194172 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:20.194151    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5"]
Sep 08 16:48:20.197573 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:20.197553    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c9b098-9bc4688f8-wq9d5"]
Sep 08 16:48:20.213242 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:20.210289    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942"]
Sep 08 16:48:20.217171 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:20.217145    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c9b098-766db8cf99-xz942"]
Sep 08 16:48:20.345396 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:20.345340    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="776168c6-1e80-46df-bf8a-b66a38f678a6" path="/var/lib/kubelet/pods/776168c6-1e80-46df-bf8a-b66a38f678a6/volumes"
Sep 08 16:48:20.345692 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:20.345679    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d" path="/var/lib/kubelet/pods/db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d/volumes"
Sep 08 16:48:20.345981 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:20.345969    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcaefa32-3362-485f-addf-7bff8ed381b8" path="/var/lib/kubelet/pods/dcaefa32-3362-485f-addf-7bff8ed381b8/volumes"
Sep 08 16:48:22.031308 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:22.031252    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k" podUID="b6d2986e-6c33-426c-bd41-eee81cdf9cfa" containerName="main" containerID="cri-o://b23828661fe4dbf8d8fa732080a246d198e193951c3b09a99bb3e821ba26eab3" gracePeriod=45
Sep 08 16:48:22.183967 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:22.183935    2557 generic.go:356] "Generic (PLEG): container finished" podID="b6d2986e-6c33-426c-bd41-eee81cdf9cfa" containerID="b23828661fe4dbf8d8fa732080a246d198e193951c3b09a99bb3e821ba26eab3" exitCode=0
Sep 08 16:48:22.184134 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:22.184007    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k" event={"ID":"b6d2986e-6c33-426c-bd41-eee81cdf9cfa","Type":"ContainerDied","Data":"b23828661fe4dbf8d8fa732080a246d198e193951c3b09a99bb3e821ba26eab3"}
Sep 08 16:48:22.294843 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:22.294773    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:48:22.371687 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:22.371659    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-tls-certs\") pod \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\" (UID: \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\") "
Sep 08 16:48:22.371861 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:22.371740    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-home\") pod \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\" (UID: \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\") "
Sep 08 16:48:22.371861 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:22.371772    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-model-cache\") pod \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\" (UID: \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\") "
Sep 08 16:48:22.371861 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:22.371789    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-dshm\") pod \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\" (UID: \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\") "
Sep 08 16:48:22.371861 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:22.371823    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wghjm\" (UniqueName: \"kubernetes.io/projected/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-kube-api-access-wghjm\") pod \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\" (UID: \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\") "
Sep 08 16:48:22.372084 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:22.371872    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-tmp-dir\") pod \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\" (UID: \"b6d2986e-6c33-426c-bd41-eee81cdf9cfa\") "
Sep 08 16:48:22.372084 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:22.371984    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-home" (OuterVolumeSpecName: "home") pod "b6d2986e-6c33-426c-bd41-eee81cdf9cfa" (UID: "b6d2986e-6c33-426c-bd41-eee81cdf9cfa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:48:22.372189 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:22.372074    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-model-cache" (OuterVolumeSpecName: "model-cache") pod "b6d2986e-6c33-426c-bd41-eee81cdf9cfa" (UID: "b6d2986e-6c33-426c-bd41-eee81cdf9cfa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:48:22.372243 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:22.372185    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b6d2986e-6c33-426c-bd41-eee81cdf9cfa" (UID: "b6d2986e-6c33-426c-bd41-eee81cdf9cfa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:48:22.372243 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:22.372192    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:22.372243 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:22.372222    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:22.374235 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:22.374205    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b6d2986e-6c33-426c-bd41-eee81cdf9cfa" (UID: "b6d2986e-6c33-426c-bd41-eee81cdf9cfa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:48:22.374235 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:22.374215    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-kube-api-access-wghjm" (OuterVolumeSpecName: "kube-api-access-wghjm") pod "b6d2986e-6c33-426c-bd41-eee81cdf9cfa" (UID: "b6d2986e-6c33-426c-bd41-eee81cdf9cfa"). InnerVolumeSpecName "kube-api-access-wghjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:48:22.374235 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:22.374211    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-dshm" (OuterVolumeSpecName: "dshm") pod "b6d2986e-6c33-426c-bd41-eee81cdf9cfa" (UID: "b6d2986e-6c33-426c-bd41-eee81cdf9cfa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:48:22.473119 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:22.473085    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:22.473119 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:22.473114    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wghjm\" (UniqueName: \"kubernetes.io/projected/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-kube-api-access-wghjm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:22.473307 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:22.473130    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:22.473307 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:22.473142    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6d2986e-6c33-426c-bd41-eee81cdf9cfa-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:23.135457 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.135401    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p" podUID="2cd532f1-a543-40d4-a6fa-cee22936725a" containerName="main" containerID="cri-o://a3b031f803e7bf38884cbe7b857d5b23d9d4605bec62dc46a174b87df17abd0d" gracePeriod=45
Sep 08 16:48:23.136427 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.136406    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:48:23.190563 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.190537    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"
Sep 08 16:48:23.190715 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.190566    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k" event={"ID":"b6d2986e-6c33-426c-bd41-eee81cdf9cfa","Type":"ContainerDied","Data":"bfd16f3e377a0ca1a93cd47d60f7d4bb3f6f7d6108a02be0478867e524311d0d"}
Sep 08 16:48:23.190715 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.190622    2557 scope.go:117] "RemoveContainer" containerID="b23828661fe4dbf8d8fa732080a246d198e193951c3b09a99bb3e821ba26eab3"
Sep 08 16:48:23.215857 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.215817    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"]
Sep 08 16:48:23.220203 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.220180    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-51b8ca/noten-model-51b8ca-kserve-6676799c8c-99j6k"]
Sep 08 16:48:23.387193 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.387137    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:48:23.480884 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.480825    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtzlt\" (UniqueName: \"kubernetes.io/projected/2cd532f1-a543-40d4-a6fa-cee22936725a-kube-api-access-dtzlt\") pod \"2cd532f1-a543-40d4-a6fa-cee22936725a\" (UID: \"2cd532f1-a543-40d4-a6fa-cee22936725a\") "
Sep 08 16:48:23.481063 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.480920    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cd532f1-a543-40d4-a6fa-cee22936725a-tls-certs\") pod \"2cd532f1-a543-40d4-a6fa-cee22936725a\" (UID: \"2cd532f1-a543-40d4-a6fa-cee22936725a\") "
Sep 08 16:48:23.481063 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.480953    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cd532f1-a543-40d4-a6fa-cee22936725a-home\") pod \"2cd532f1-a543-40d4-a6fa-cee22936725a\" (UID: \"2cd532f1-a543-40d4-a6fa-cee22936725a\") "
Sep 08 16:48:23.481063 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.480979    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cd532f1-a543-40d4-a6fa-cee22936725a-model-cache\") pod \"2cd532f1-a543-40d4-a6fa-cee22936725a\" (UID: \"2cd532f1-a543-40d4-a6fa-cee22936725a\") "
Sep 08 16:48:23.481063 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.481010    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cd532f1-a543-40d4-a6fa-cee22936725a-tmp-dir\") pod \"2cd532f1-a543-40d4-a6fa-cee22936725a\" (UID: \"2cd532f1-a543-40d4-a6fa-cee22936725a\") "
Sep 08 16:48:23.481063 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.481061    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cd532f1-a543-40d4-a6fa-cee22936725a-dshm\") pod \"2cd532f1-a543-40d4-a6fa-cee22936725a\" (UID: \"2cd532f1-a543-40d4-a6fa-cee22936725a\") "
Sep 08 16:48:23.481338 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.481225    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cd532f1-a543-40d4-a6fa-cee22936725a-home" (OuterVolumeSpecName: "home") pod "2cd532f1-a543-40d4-a6fa-cee22936725a" (UID: "2cd532f1-a543-40d4-a6fa-cee22936725a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:48:23.481338 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.481237    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cd532f1-a543-40d4-a6fa-cee22936725a-model-cache" (OuterVolumeSpecName: "model-cache") pod "2cd532f1-a543-40d4-a6fa-cee22936725a" (UID: "2cd532f1-a543-40d4-a6fa-cee22936725a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:48:23.481338 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.481319    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cd532f1-a543-40d4-a6fa-cee22936725a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2cd532f1-a543-40d4-a6fa-cee22936725a" (UID: "2cd532f1-a543-40d4-a6fa-cee22936725a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:48:23.483258 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.483236    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cd532f1-a543-40d4-a6fa-cee22936725a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2cd532f1-a543-40d4-a6fa-cee22936725a" (UID: "2cd532f1-a543-40d4-a6fa-cee22936725a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:48:23.483746 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.483729    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cd532f1-a543-40d4-a6fa-cee22936725a-dshm" (OuterVolumeSpecName: "dshm") pod "2cd532f1-a543-40d4-a6fa-cee22936725a" (UID: "2cd532f1-a543-40d4-a6fa-cee22936725a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:48:23.483746 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.483740    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd532f1-a543-40d4-a6fa-cee22936725a-kube-api-access-dtzlt" (OuterVolumeSpecName: "kube-api-access-dtzlt") pod "2cd532f1-a543-40d4-a6fa-cee22936725a" (UID: "2cd532f1-a543-40d4-a6fa-cee22936725a"). InnerVolumeSpecName "kube-api-access-dtzlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:48:23.582157 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.582127    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtzlt\" (UniqueName: \"kubernetes.io/projected/2cd532f1-a543-40d4-a6fa-cee22936725a-kube-api-access-dtzlt\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:23.582157 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.582154    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cd532f1-a543-40d4-a6fa-cee22936725a-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:23.582157 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.582165    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cd532f1-a543-40d4-a6fa-cee22936725a-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:23.582388 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.582173    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cd532f1-a543-40d4-a6fa-cee22936725a-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:23.582388 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.582182    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cd532f1-a543-40d4-a6fa-cee22936725a-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:23.582388 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:23.582194    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cd532f1-a543-40d4-a6fa-cee22936725a-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:24.197413 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:24.197376    2557 generic.go:356] "Generic (PLEG): container finished" podID="2cd532f1-a543-40d4-a6fa-cee22936725a" containerID="a3b031f803e7bf38884cbe7b857d5b23d9d4605bec62dc46a174b87df17abd0d" exitCode=0
Sep 08 16:48:24.197815 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:24.197422    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"
Sep 08 16:48:24.197815 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:24.197469    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p" event={"ID":"2cd532f1-a543-40d4-a6fa-cee22936725a","Type":"ContainerDied","Data":"a3b031f803e7bf38884cbe7b857d5b23d9d4605bec62dc46a174b87df17abd0d"}
Sep 08 16:48:24.197815 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:24.197511    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p" event={"ID":"2cd532f1-a543-40d4-a6fa-cee22936725a","Type":"ContainerDied","Data":"9d15ead713890b89887bc582ce49e587edcd3933f15c23343e7dc29cd4374d3e"}
Sep 08 16:48:24.197815 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:24.197529    2557 scope.go:117] "RemoveContainer" containerID="a3b031f803e7bf38884cbe7b857d5b23d9d4605bec62dc46a174b87df17abd0d"
Sep 08 16:48:24.206996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:24.206980    2557 scope.go:117] "RemoveContainer" containerID="a3b031f803e7bf38884cbe7b857d5b23d9d4605bec62dc46a174b87df17abd0d"
Sep 08 16:48:24.207253 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:48:24.207232    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3b031f803e7bf38884cbe7b857d5b23d9d4605bec62dc46a174b87df17abd0d\": container with ID starting with a3b031f803e7bf38884cbe7b857d5b23d9d4605bec62dc46a174b87df17abd0d not found: ID does not exist" containerID="a3b031f803e7bf38884cbe7b857d5b23d9d4605bec62dc46a174b87df17abd0d"
Sep 08 16:48:24.207352 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:24.207258    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3b031f803e7bf38884cbe7b857d5b23d9d4605bec62dc46a174b87df17abd0d"} err="failed to get container status \"a3b031f803e7bf38884cbe7b857d5b23d9d4605bec62dc46a174b87df17abd0d\": rpc error: code = NotFound desc = could not find container \"a3b031f803e7bf38884cbe7b857d5b23d9d4605bec62dc46a174b87df17abd0d\": container with ID starting with a3b031f803e7bf38884cbe7b857d5b23d9d4605bec62dc46a174b87df17abd0d not found: ID does not exist"
Sep 08 16:48:24.218665 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:24.218642    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"]
Sep 08 16:48:24.222536 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:24.222513    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-43a905/test-model-43a905-kserve-5bd5f577df-6lq9p"]
Sep 08 16:48:24.345525 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:24.345497    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cd532f1-a543-40d4-a6fa-cee22936725a" path="/var/lib/kubelet/pods/2cd532f1-a543-40d4-a6fa-cee22936725a/volumes"
Sep 08 16:48:24.345962 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:24.345943    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6d2986e-6c33-426c-bd41-eee81cdf9cfa" path="/var/lib/kubelet/pods/b6d2986e-6c33-426c-bd41-eee81cdf9cfa/volumes"
Sep 08 16:48:25.651242 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:25.651212    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm"]
Sep 08 16:48:25.651688 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:25.651424    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm" podUID="70bf6456-34db-4039-8d3c-21b1355a21c2" containerName="payload-processing" containerID="cri-o://f186a539a8e9cb366998d74d22bfbe7acff9af5a80a7ea09970ae94e2c5ba6b8" gracePeriod=30
Sep 08 16:48:25.658963 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:25.658936    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx"]
Sep 08 16:48:25.659166 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:25.659131    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx" podUID="c3f1ca04-44b4-48b1-9fb7-3da2f8c29814" containerName="payload-pre-processing" containerID="cri-o://19c9677d3adf53d834ddbf61cdc4c685ebbbb1d61a5c71a1dcdae36933027147" gracePeriod=30
Sep 08 16:48:25.936015 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:25.935991    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdae4bb7-jmzhz"]
Sep 08 16:48:25.938696 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:25.938675    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bdae4bb7-jmzhz"]
Sep 08 16:48:25.947967 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:25.947948    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm"
Sep 08 16:48:25.950568 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:25.950545    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx"
Sep 08 16:48:26.000014 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:25.999968    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70bf6456-34db-4039-8d3c-21b1355a21c2-plugins-config-volume\") pod \"70bf6456-34db-4039-8d3c-21b1355a21c2\" (UID: \"70bf6456-34db-4039-8d3c-21b1355a21c2\") "
Sep 08 16:48:26.000182 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.000084    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-584bh\" (UniqueName: \"kubernetes.io/projected/70bf6456-34db-4039-8d3c-21b1355a21c2-kube-api-access-584bh\") pod \"70bf6456-34db-4039-8d3c-21b1355a21c2\" (UID: \"70bf6456-34db-4039-8d3c-21b1355a21c2\") "
Sep 08 16:48:26.000311 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.000289    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70bf6456-34db-4039-8d3c-21b1355a21c2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70bf6456-34db-4039-8d3c-21b1355a21c2" (UID: "70bf6456-34db-4039-8d3c-21b1355a21c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:48:26.002234 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.002208    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bf6456-34db-4039-8d3c-21b1355a21c2-kube-api-access-584bh" (OuterVolumeSpecName: "kube-api-access-584bh") pod "70bf6456-34db-4039-8d3c-21b1355a21c2" (UID: "70bf6456-34db-4039-8d3c-21b1355a21c2"). InnerVolumeSpecName "kube-api-access-584bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:48:26.101109 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.101070    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgcz4\" (UniqueName: \"kubernetes.io/projected/c3f1ca04-44b4-48b1-9fb7-3da2f8c29814-kube-api-access-xgcz4\") pod \"c3f1ca04-44b4-48b1-9fb7-3da2f8c29814\" (UID: \"c3f1ca04-44b4-48b1-9fb7-3da2f8c29814\") "
Sep 08 16:48:26.101239 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.101163    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f1ca04-44b4-48b1-9fb7-3da2f8c29814-plugins-config-volume\") pod \"c3f1ca04-44b4-48b1-9fb7-3da2f8c29814\" (UID: \"c3f1ca04-44b4-48b1-9fb7-3da2f8c29814\") "
Sep 08 16:48:26.101346 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.101329    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70bf6456-34db-4039-8d3c-21b1355a21c2-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:26.101390 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.101352    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-584bh\" (UniqueName: \"kubernetes.io/projected/70bf6456-34db-4039-8d3c-21b1355a21c2-kube-api-access-584bh\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:26.101528 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.101506    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3f1ca04-44b4-48b1-9fb7-3da2f8c29814-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c3f1ca04-44b4-48b1-9fb7-3da2f8c29814" (UID: "c3f1ca04-44b4-48b1-9fb7-3da2f8c29814"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:48:26.103237 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.103215    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3f1ca04-44b4-48b1-9fb7-3da2f8c29814-kube-api-access-xgcz4" (OuterVolumeSpecName: "kube-api-access-xgcz4") pod "c3f1ca04-44b4-48b1-9fb7-3da2f8c29814" (UID: "c3f1ca04-44b4-48b1-9fb7-3da2f8c29814"). InnerVolumeSpecName "kube-api-access-xgcz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:48:26.202356 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.202287    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgcz4\" (UniqueName: \"kubernetes.io/projected/c3f1ca04-44b4-48b1-9fb7-3da2f8c29814-kube-api-access-xgcz4\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:26.202356 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.202314    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f1ca04-44b4-48b1-9fb7-3da2f8c29814-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:26.216747 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.216720    2557 generic.go:356] "Generic (PLEG): container finished" podID="c3f1ca04-44b4-48b1-9fb7-3da2f8c29814" containerID="19c9677d3adf53d834ddbf61cdc4c685ebbbb1d61a5c71a1dcdae36933027147" exitCode=0
Sep 08 16:48:26.216906 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.216763    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx"
Sep 08 16:48:26.216906 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.216813    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx" event={"ID":"c3f1ca04-44b4-48b1-9fb7-3da2f8c29814","Type":"ContainerDied","Data":"19c9677d3adf53d834ddbf61cdc4c685ebbbb1d61a5c71a1dcdae36933027147"}
Sep 08 16:48:26.216906 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.216882    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx" event={"ID":"c3f1ca04-44b4-48b1-9fb7-3da2f8c29814","Type":"ContainerDied","Data":"dc67dca8f095b7baba07b120f353ea4123751a6f8d910ca39a9f198b4907996f"}
Sep 08 16:48:26.217069 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.216910    2557 scope.go:117] "RemoveContainer" containerID="19c9677d3adf53d834ddbf61cdc4c685ebbbb1d61a5c71a1dcdae36933027147"
Sep 08 16:48:26.218170 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.218147    2557 generic.go:356] "Generic (PLEG): container finished" podID="70bf6456-34db-4039-8d3c-21b1355a21c2" containerID="f186a539a8e9cb366998d74d22bfbe7acff9af5a80a7ea09970ae94e2c5ba6b8" exitCode=0
Sep 08 16:48:26.218244 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.218225    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm" event={"ID":"70bf6456-34db-4039-8d3c-21b1355a21c2","Type":"ContainerDied","Data":"f186a539a8e9cb366998d74d22bfbe7acff9af5a80a7ea09970ae94e2c5ba6b8"}
Sep 08 16:48:26.218302 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.218257    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm" event={"ID":"70bf6456-34db-4039-8d3c-21b1355a21c2","Type":"ContainerDied","Data":"56b62228e79c1e81582f36145ced82872f49d621a25d294604a4de5d0bae5f50"}
Sep 08 16:48:26.218302 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.218227    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm"
Sep 08 16:48:26.225743 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.225730    2557 scope.go:117] "RemoveContainer" containerID="19c9677d3adf53d834ddbf61cdc4c685ebbbb1d61a5c71a1dcdae36933027147"
Sep 08 16:48:26.226055 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:48:26.226024    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19c9677d3adf53d834ddbf61cdc4c685ebbbb1d61a5c71a1dcdae36933027147\": container with ID starting with 19c9677d3adf53d834ddbf61cdc4c685ebbbb1d61a5c71a1dcdae36933027147 not found: ID does not exist" containerID="19c9677d3adf53d834ddbf61cdc4c685ebbbb1d61a5c71a1dcdae36933027147"
Sep 08 16:48:26.226257 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.226062    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c9677d3adf53d834ddbf61cdc4c685ebbbb1d61a5c71a1dcdae36933027147"} err="failed to get container status \"19c9677d3adf53d834ddbf61cdc4c685ebbbb1d61a5c71a1dcdae36933027147\": rpc error: code = NotFound desc = could not find container \"19c9677d3adf53d834ddbf61cdc4c685ebbbb1d61a5c71a1dcdae36933027147\": container with ID starting with 19c9677d3adf53d834ddbf61cdc4c685ebbbb1d61a5c71a1dcdae36933027147 not found: ID does not exist"
Sep 08 16:48:26.226257 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.226084    2557 scope.go:117] "RemoveContainer" containerID="f186a539a8e9cb366998d74d22bfbe7acff9af5a80a7ea09970ae94e2c5ba6b8"
Sep 08 16:48:26.236908 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.236891    2557 scope.go:117] "RemoveContainer" containerID="f186a539a8e9cb366998d74d22bfbe7acff9af5a80a7ea09970ae94e2c5ba6b8"
Sep 08 16:48:26.237170 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:48:26.237152    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f186a539a8e9cb366998d74d22bfbe7acff9af5a80a7ea09970ae94e2c5ba6b8\": container with ID starting with f186a539a8e9cb366998d74d22bfbe7acff9af5a80a7ea09970ae94e2c5ba6b8 not found: ID does not exist" containerID="f186a539a8e9cb366998d74d22bfbe7acff9af5a80a7ea09970ae94e2c5ba6b8"
Sep 08 16:48:26.237236 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.237180    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f186a539a8e9cb366998d74d22bfbe7acff9af5a80a7ea09970ae94e2c5ba6b8"} err="failed to get container status \"f186a539a8e9cb366998d74d22bfbe7acff9af5a80a7ea09970ae94e2c5ba6b8\": rpc error: code = NotFound desc = could not find container \"f186a539a8e9cb366998d74d22bfbe7acff9af5a80a7ea09970ae94e2c5ba6b8\": container with ID starting with f186a539a8e9cb366998d74d22bfbe7acff9af5a80a7ea09970ae94e2c5ba6b8 not found: ID does not exist"
Sep 08 16:48:26.239672 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.239648    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx"]
Sep 08 16:48:26.243776 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.243756    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bdae4bb7-75df76c898-xkgkx"]
Sep 08 16:48:26.254748 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.254725    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm"]
Sep 08 16:48:26.256647 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.256629    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bdae4bb7-79c8fc9b45-t7mjm"]
Sep 08 16:48:26.344878 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.344825    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15268038-3db7-4ef7-9945-81f80f74d4a8" path="/var/lib/kubelet/pods/15268038-3db7-4ef7-9945-81f80f74d4a8/volumes"
Sep 08 16:48:26.345246 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.345228    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70bf6456-34db-4039-8d3c-21b1355a21c2" path="/var/lib/kubelet/pods/70bf6456-34db-4039-8d3c-21b1355a21c2/volumes"
Sep 08 16:48:26.345573 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.345557    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3f1ca04-44b4-48b1-9fb7-3da2f8c29814" path="/var/lib/kubelet/pods/c3f1ca04-44b4-48b1-9fb7-3da2f8c29814/volumes"
Sep 08 16:48:26.869065 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.869036    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-43a905-w5k7r"]
Sep 08 16:48:26.869394 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.869338    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d" containerName="payload-pre-processing"
Sep 08 16:48:26.869394 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.869349    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d" containerName="payload-pre-processing"
Sep 08 16:48:26.869394 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.869357    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="776168c6-1e80-46df-bf8a-b66a38f678a6" containerName="payload-processing"
Sep 08 16:48:26.869394 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.869363    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="776168c6-1e80-46df-bf8a-b66a38f678a6" containerName="payload-processing"
Sep 08 16:48:26.869394 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.869369    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cd532f1-a543-40d4-a6fa-cee22936725a" containerName="main"
Sep 08 16:48:26.869394 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.869374    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cd532f1-a543-40d4-a6fa-cee22936725a" containerName="main"
Sep 08 16:48:26.869394 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.869386    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15268038-3db7-4ef7-9945-81f80f74d4a8" containerName="revoke-keys"
Sep 08 16:48:26.869394 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.869391    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="15268038-3db7-4ef7-9945-81f80f74d4a8" containerName="revoke-keys"
Sep 08 16:48:26.869394 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.869398    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3f1ca04-44b4-48b1-9fb7-3da2f8c29814" containerName="payload-pre-processing"
Sep 08 16:48:26.869650 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.869403    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3f1ca04-44b4-48b1-9fb7-3da2f8c29814" containerName="payload-pre-processing"
Sep 08 16:48:26.869650 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.869412    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70bf6456-34db-4039-8d3c-21b1355a21c2" containerName="payload-processing"
Sep 08 16:48:26.869650 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.869417    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="70bf6456-34db-4039-8d3c-21b1355a21c2" containerName="payload-processing"
Sep 08 16:48:26.869650 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.869426    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6d2986e-6c33-426c-bd41-eee81cdf9cfa" containerName="main"
Sep 08 16:48:26.869650 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.869430    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6d2986e-6c33-426c-bd41-eee81cdf9cfa" containerName="main"
Sep 08 16:48:26.869650 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.869475    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="776168c6-1e80-46df-bf8a-b66a38f678a6" containerName="payload-processing"
Sep 08 16:48:26.869650 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.869482    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="db2ce4c5-0c6f-4c5e-9929-6bc44d3c980d" containerName="payload-pre-processing"
Sep 08 16:48:26.869650 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.869491    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="15268038-3db7-4ef7-9945-81f80f74d4a8" containerName="revoke-keys"
Sep 08 16:48:26.869650 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.869496    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3f1ca04-44b4-48b1-9fb7-3da2f8c29814" containerName="payload-pre-processing"
Sep 08 16:48:26.869650 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.869501    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="70bf6456-34db-4039-8d3c-21b1355a21c2" containerName="payload-processing"
Sep 08 16:48:26.869650 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.869508    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cd532f1-a543-40d4-a6fa-cee22936725a" containerName="main"
Sep 08 16:48:26.869650 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.869512    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6d2986e-6c33-426c-bd41-eee81cdf9cfa" containerName="main"
Sep 08 16:48:26.877530 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.877502    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-43a905-w5k7r"
Sep 08 16:48:26.879638 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.879612    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-43a905-w5k7r"]
Sep 08 16:48:26.879897 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.879880    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:48:26.879978 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:26.879884    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-886jf\""
Sep 08 16:48:27.008000 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:27.007928    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0df51c4-a808-4be0-aeb4-73d24da47b54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-43a905-w5k7r\" (UID: \"d0df51c4-a808-4be0-aeb4-73d24da47b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-43a905-w5k7r"
Sep 08 16:48:27.108467 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:27.108436    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0df51c4-a808-4be0-aeb4-73d24da47b54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-43a905-w5k7r\" (UID: \"d0df51c4-a808-4be0-aeb4-73d24da47b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-43a905-w5k7r"
Sep 08 16:48:27.109055 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:27.109036    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0df51c4-a808-4be0-aeb4-73d24da47b54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-43a905-w5k7r\" (UID: \"d0df51c4-a808-4be0-aeb4-73d24da47b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-43a905-w5k7r"
Sep 08 16:48:27.189816 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:27.189788    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-43a905-w5k7r"
Sep 08 16:48:27.459110 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:27.459089    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-43a905-w5k7r"]
Sep 08 16:48:27.460982 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:48:27.460962    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0df51c4_a808_4be0_aeb4_73d24da47b54.slice/crio-a2332504c24778bce33caaf582853085df61a07b926954e4fe23210937fee9c9 WatchSource:0}: Error finding container a2332504c24778bce33caaf582853085df61a07b926954e4fe23210937fee9c9: Status 404 returned error can't find the container with id a2332504c24778bce33caaf582853085df61a07b926954e4fe23210937fee9c9
Sep 08 16:48:28.232760 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:28.232734    2557 generic.go:356] "Generic (PLEG): container finished" podID="d0df51c4-a808-4be0-aeb4-73d24da47b54" containerID="06167e2c3d0a044c20d017a9de0a08d347a350032edc9c9223a50de44d502030" exitCode=0
Sep 08 16:48:28.233067 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:28.232795    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-43a905-w5k7r" event={"ID":"d0df51c4-a808-4be0-aeb4-73d24da47b54","Type":"ContainerDied","Data":"06167e2c3d0a044c20d017a9de0a08d347a350032edc9c9223a50de44d502030"}
Sep 08 16:48:28.233067 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:28.232850    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-43a905-w5k7r" event={"ID":"d0df51c4-a808-4be0-aeb4-73d24da47b54","Type":"ContainerStarted","Data":"a2332504c24778bce33caaf582853085df61a07b926954e4fe23210937fee9c9"}
Sep 08 16:48:29.366526 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:29.366503    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-43a905-w5k7r"
Sep 08 16:48:29.425120 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:29.425088    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0df51c4-a808-4be0-aeb4-73d24da47b54-maas-api-service-ca\") pod \"d0df51c4-a808-4be0-aeb4-73d24da47b54\" (UID: \"d0df51c4-a808-4be0-aeb4-73d24da47b54\") "
Sep 08 16:48:29.425397 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:29.425376    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0df51c4-a808-4be0-aeb4-73d24da47b54-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d0df51c4-a808-4be0-aeb4-73d24da47b54" (UID: "d0df51c4-a808-4be0-aeb4-73d24da47b54"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:48:29.525933 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:29.525871    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0df51c4-a808-4be0-aeb4-73d24da47b54-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:30.243696 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:30.243669    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-43a905-w5k7r"
Sep 08 16:48:30.243994 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:30.243699    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-43a905-w5k7r" event={"ID":"d0df51c4-a808-4be0-aeb4-73d24da47b54","Type":"ContainerDied","Data":"a2332504c24778bce33caaf582853085df61a07b926954e4fe23210937fee9c9"}
Sep 08 16:48:30.243994 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:30.243768    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2332504c24778bce33caaf582853085df61a07b926954e4fe23210937fee9c9"
Sep 08 16:48:31.940385 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:31.940350    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:48:37.360764 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:37.360734    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-43a905-29814765-b6jrv"]
Sep 08 16:48:37.365362 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:37.365338    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-43a905-29814765-b6jrv"]
Sep 08 16:48:37.389541 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:37.389510    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd"]
Sep 08 16:48:37.390051 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:37.390015    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd" podUID="789ae719-5721-4edd-b05f-b5a312b3ec4c" containerName="payload-processing" containerID="cri-o://e578d457afad87178a29bfd65e992cc46da691f6130e4001749310815f828932" gracePeriod=30
Sep 08 16:48:37.396681 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:37.396660    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt"]
Sep 08 16:48:37.397076 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:37.396997    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt" podUID="b0f3c2c9-b701-4d0e-b5ce-047682111980" containerName="payload-pre-processing" containerID="cri-o://f091aefdafff496abe0d6fc1516622f0ad9fcb4430e35feb6381e085da2760c7" gracePeriod=30
Sep 08 16:48:37.561235 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:37.561203    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-43a905-w5k7r"]
Sep 08 16:48:37.564429 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:37.564375    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-43a905-w5k7r"]
Sep 08 16:48:37.663778 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:37.663758    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd"
Sep 08 16:48:37.666295 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:37.666278    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt"
Sep 08 16:48:37.788883 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:37.788853    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789ae719-5721-4edd-b05f-b5a312b3ec4c-plugins-config-volume\") pod \"789ae719-5721-4edd-b05f-b5a312b3ec4c\" (UID: \"789ae719-5721-4edd-b05f-b5a312b3ec4c\") "
Sep 08 16:48:37.789022 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:37.788890    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48frh\" (UniqueName: \"kubernetes.io/projected/789ae719-5721-4edd-b05f-b5a312b3ec4c-kube-api-access-48frh\") pod \"789ae719-5721-4edd-b05f-b5a312b3ec4c\" (UID: \"789ae719-5721-4edd-b05f-b5a312b3ec4c\") "
Sep 08 16:48:37.789022 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:37.788946    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f3c2c9-b701-4d0e-b5ce-047682111980-plugins-config-volume\") pod \"b0f3c2c9-b701-4d0e-b5ce-047682111980\" (UID: \"b0f3c2c9-b701-4d0e-b5ce-047682111980\") "
Sep 08 16:48:37.789022 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:37.788985    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ql88\" (UniqueName: \"kubernetes.io/projected/b0f3c2c9-b701-4d0e-b5ce-047682111980-kube-api-access-4ql88\") pod \"b0f3c2c9-b701-4d0e-b5ce-047682111980\" (UID: \"b0f3c2c9-b701-4d0e-b5ce-047682111980\") "
Sep 08 16:48:37.789265 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:37.789244    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789ae719-5721-4edd-b05f-b5a312b3ec4c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "789ae719-5721-4edd-b05f-b5a312b3ec4c" (UID: "789ae719-5721-4edd-b05f-b5a312b3ec4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:48:37.789343 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:37.789318    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0f3c2c9-b701-4d0e-b5ce-047682111980-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b0f3c2c9-b701-4d0e-b5ce-047682111980" (UID: "b0f3c2c9-b701-4d0e-b5ce-047682111980"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:48:37.791193 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:37.791171    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789ae719-5721-4edd-b05f-b5a312b3ec4c-kube-api-access-48frh" (OuterVolumeSpecName: "kube-api-access-48frh") pod "789ae719-5721-4edd-b05f-b5a312b3ec4c" (UID: "789ae719-5721-4edd-b05f-b5a312b3ec4c"). InnerVolumeSpecName "kube-api-access-48frh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:48:37.791299 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:37.791236    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f3c2c9-b701-4d0e-b5ce-047682111980-kube-api-access-4ql88" (OuterVolumeSpecName: "kube-api-access-4ql88") pod "b0f3c2c9-b701-4d0e-b5ce-047682111980" (UID: "b0f3c2c9-b701-4d0e-b5ce-047682111980"). InnerVolumeSpecName "kube-api-access-4ql88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:48:37.889760 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:37.889735    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f3c2c9-b701-4d0e-b5ce-047682111980-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:37.889760 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:37.889758    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ql88\" (UniqueName: \"kubernetes.io/projected/b0f3c2c9-b701-4d0e-b5ce-047682111980-kube-api-access-4ql88\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:37.889928 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:37.889767    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789ae719-5721-4edd-b05f-b5a312b3ec4c-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:37.889928 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:37.889777    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48frh\" (UniqueName: \"kubernetes.io/projected/789ae719-5721-4edd-b05f-b5a312b3ec4c-kube-api-access-48frh\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:48:38.285785 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:38.285754    2557 generic.go:356] "Generic (PLEG): container finished" podID="b0f3c2c9-b701-4d0e-b5ce-047682111980" containerID="f091aefdafff496abe0d6fc1516622f0ad9fcb4430e35feb6381e085da2760c7" exitCode=0
Sep 08 16:48:38.285989 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:38.285808    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt"
Sep 08 16:48:38.285989 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:38.285856    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt" event={"ID":"b0f3c2c9-b701-4d0e-b5ce-047682111980","Type":"ContainerDied","Data":"f091aefdafff496abe0d6fc1516622f0ad9fcb4430e35feb6381e085da2760c7"}
Sep 08 16:48:38.285989 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:38.285913    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt" event={"ID":"b0f3c2c9-b701-4d0e-b5ce-047682111980","Type":"ContainerDied","Data":"6a7bf5ae51e306377b60695fe92a5454cadba3bb540618ca497c626df7708fa4"}
Sep 08 16:48:38.285989 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:38.285936    2557 scope.go:117] "RemoveContainer" containerID="f091aefdafff496abe0d6fc1516622f0ad9fcb4430e35feb6381e085da2760c7"
Sep 08 16:48:38.287281 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:38.287260    2557 generic.go:356] "Generic (PLEG): container finished" podID="789ae719-5721-4edd-b05f-b5a312b3ec4c" containerID="e578d457afad87178a29bfd65e992cc46da691f6130e4001749310815f828932" exitCode=0
Sep 08 16:48:38.287394 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:38.287306    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd"
Sep 08 16:48:38.287394 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:38.287305    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd" event={"ID":"789ae719-5721-4edd-b05f-b5a312b3ec4c","Type":"ContainerDied","Data":"e578d457afad87178a29bfd65e992cc46da691f6130e4001749310815f828932"}
Sep 08 16:48:38.287497 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:38.287409    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd" event={"ID":"789ae719-5721-4edd-b05f-b5a312b3ec4c","Type":"ContainerDied","Data":"b94a20cb5f61a2ddda6d11c36d8383025f9e661eb38ac56c901e643040977c40"}
Sep 08 16:48:38.295640 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:38.295621    2557 scope.go:117] "RemoveContainer" containerID="f091aefdafff496abe0d6fc1516622f0ad9fcb4430e35feb6381e085da2760c7"
Sep 08 16:48:38.295924 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:48:38.295899    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f091aefdafff496abe0d6fc1516622f0ad9fcb4430e35feb6381e085da2760c7\": container with ID starting with f091aefdafff496abe0d6fc1516622f0ad9fcb4430e35feb6381e085da2760c7 not found: ID does not exist" containerID="f091aefdafff496abe0d6fc1516622f0ad9fcb4430e35feb6381e085da2760c7"
Sep 08 16:48:38.296031 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:38.295927    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f091aefdafff496abe0d6fc1516622f0ad9fcb4430e35feb6381e085da2760c7"} err="failed to get container status \"f091aefdafff496abe0d6fc1516622f0ad9fcb4430e35feb6381e085da2760c7\": rpc error: code = NotFound desc = could not find container \"f091aefdafff496abe0d6fc1516622f0ad9fcb4430e35feb6381e085da2760c7\": container with ID starting with f091aefdafff496abe0d6fc1516622f0ad9fcb4430e35feb6381e085da2760c7 not found: ID does not exist"
Sep 08 16:48:38.296031 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:38.295943    2557 scope.go:117] "RemoveContainer" containerID="e578d457afad87178a29bfd65e992cc46da691f6130e4001749310815f828932"
Sep 08 16:48:38.303660 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:38.303647    2557 scope.go:117] "RemoveContainer" containerID="e578d457afad87178a29bfd65e992cc46da691f6130e4001749310815f828932"
Sep 08 16:48:38.303937 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:48:38.303922    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e578d457afad87178a29bfd65e992cc46da691f6130e4001749310815f828932\": container with ID starting with e578d457afad87178a29bfd65e992cc46da691f6130e4001749310815f828932 not found: ID does not exist" containerID="e578d457afad87178a29bfd65e992cc46da691f6130e4001749310815f828932"
Sep 08 16:48:38.303988 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:38.303944    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e578d457afad87178a29bfd65e992cc46da691f6130e4001749310815f828932"} err="failed to get container status \"e578d457afad87178a29bfd65e992cc46da691f6130e4001749310815f828932\": rpc error: code = NotFound desc = could not find container \"e578d457afad87178a29bfd65e992cc46da691f6130e4001749310815f828932\": container with ID starting with e578d457afad87178a29bfd65e992cc46da691f6130e4001749310815f828932 not found: ID does not exist"
Sep 08 16:48:38.310644 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:38.310624    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd"]
Sep 08 16:48:38.313262 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:38.313243    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-43a905-7fd6f75b4c-xllsd"]
Sep 08 16:48:38.323489 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:38.323471    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt"]
Sep 08 16:48:38.332458 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:38.332438    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-43a905-76dcc6746-2vvwt"]
Sep 08 16:48:38.345139 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:38.345118    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="789ae719-5721-4edd-b05f-b5a312b3ec4c" path="/var/lib/kubelet/pods/789ae719-5721-4edd-b05f-b5a312b3ec4c/volumes"
Sep 08 16:48:38.345418 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:38.345406    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0f3c2c9-b701-4d0e-b5ce-047682111980" path="/var/lib/kubelet/pods/b0f3c2c9-b701-4d0e-b5ce-047682111980/volumes"
Sep 08 16:48:38.345618 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:38.345608    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c153a0c2-da40-41a1-944c-9a7e3df669b8" path="/var/lib/kubelet/pods/c153a0c2-da40-41a1-944c-9a7e3df669b8/volumes"
Sep 08 16:48:38.345828 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:38.345817    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0df51c4-a808-4be0-aeb4-73d24da47b54" path="/var/lib/kubelet/pods/d0df51c4-a808-4be0-aeb4-73d24da47b54/volumes"
Sep 08 16:48:40.428493 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:40.428464    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:48:50.381003 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.380963    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg"]
Sep 08 16:48:50.381463 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.381407    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0df51c4-a808-4be0-aeb4-73d24da47b54" containerName="revoke-keys"
Sep 08 16:48:50.381463 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.381424    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0df51c4-a808-4be0-aeb4-73d24da47b54" containerName="revoke-keys"
Sep 08 16:48:50.381463 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.381434    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0f3c2c9-b701-4d0e-b5ce-047682111980" containerName="payload-pre-processing"
Sep 08 16:48:50.381463 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.381442    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0f3c2c9-b701-4d0e-b5ce-047682111980" containerName="payload-pre-processing"
Sep 08 16:48:50.381667 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.381471    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="789ae719-5721-4edd-b05f-b5a312b3ec4c" containerName="payload-processing"
Sep 08 16:48:50.381667 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.381479    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="789ae719-5721-4edd-b05f-b5a312b3ec4c" containerName="payload-processing"
Sep 08 16:48:50.381667 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.381554    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0f3c2c9-b701-4d0e-b5ce-047682111980" containerName="payload-pre-processing"
Sep 08 16:48:50.381667 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.381568    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0df51c4-a808-4be0-aeb4-73d24da47b54" containerName="revoke-keys"
Sep 08 16:48:50.381667 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.381577    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="789ae719-5721-4edd-b05f-b5a312b3ec4c" containerName="payload-processing"
Sep 08 16:48:50.384588 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.384567    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg"
Sep 08 16:48:50.387126 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.387109    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-608268-dockercfg-q694p\""
Sep 08 16:48:50.387203 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.387152    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-608268\""
Sep 08 16:48:50.397786 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.397767    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg"]
Sep 08 16:48:50.416166 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.416146    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh"]
Sep 08 16:48:50.419346 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.419329    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh"
Sep 08 16:48:50.431922 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.431900    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh"]
Sep 08 16:48:50.497060 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.497024    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rh8q\" (UniqueName: \"kubernetes.io/projected/4e0f270f-0dcb-4261-baea-8a77e154f53b-kube-api-access-7rh8q\") pod \"payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg\" (UID: \"4e0f270f-0dcb-4261-baea-8a77e154f53b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg"
Sep 08 16:48:50.497231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.497084    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e0f270f-0dcb-4261-baea-8a77e154f53b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg\" (UID: \"4e0f270f-0dcb-4261-baea-8a77e154f53b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg"
Sep 08 16:48:50.598130 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.598086    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c90ce47-9dbe-42f4-b9a9-a448567a00d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh\" (UID: \"4c90ce47-9dbe-42f4-b9a9-a448567a00d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh"
Sep 08 16:48:50.598130 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.598121    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdkxk\" (UniqueName: \"kubernetes.io/projected/4c90ce47-9dbe-42f4-b9a9-a448567a00d7-kube-api-access-wdkxk\") pod \"payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh\" (UID: \"4c90ce47-9dbe-42f4-b9a9-a448567a00d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh"
Sep 08 16:48:50.598359 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.598164    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rh8q\" (UniqueName: \"kubernetes.io/projected/4e0f270f-0dcb-4261-baea-8a77e154f53b-kube-api-access-7rh8q\") pod \"payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg\" (UID: \"4e0f270f-0dcb-4261-baea-8a77e154f53b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg"
Sep 08 16:48:50.598359 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.598256    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e0f270f-0dcb-4261-baea-8a77e154f53b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg\" (UID: \"4e0f270f-0dcb-4261-baea-8a77e154f53b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg"
Sep 08 16:48:50.598950 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.598933    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e0f270f-0dcb-4261-baea-8a77e154f53b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg\" (UID: \"4e0f270f-0dcb-4261-baea-8a77e154f53b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg"
Sep 08 16:48:50.606803 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.606784    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rh8q\" (UniqueName: \"kubernetes.io/projected/4e0f270f-0dcb-4261-baea-8a77e154f53b-kube-api-access-7rh8q\") pod \"payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg\" (UID: \"4e0f270f-0dcb-4261-baea-8a77e154f53b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg"
Sep 08 16:48:50.694277 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.694250    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg"
Sep 08 16:48:50.699580 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.699536    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c90ce47-9dbe-42f4-b9a9-a448567a00d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh\" (UID: \"4c90ce47-9dbe-42f4-b9a9-a448567a00d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh"
Sep 08 16:48:50.699668 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.699595    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdkxk\" (UniqueName: \"kubernetes.io/projected/4c90ce47-9dbe-42f4-b9a9-a448567a00d7-kube-api-access-wdkxk\") pod \"payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh\" (UID: \"4c90ce47-9dbe-42f4-b9a9-a448567a00d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh"
Sep 08 16:48:50.700289 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.700251    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c90ce47-9dbe-42f4-b9a9-a448567a00d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh\" (UID: \"4c90ce47-9dbe-42f4-b9a9-a448567a00d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh"
Sep 08 16:48:50.710705 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.710684    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdkxk\" (UniqueName: \"kubernetes.io/projected/4c90ce47-9dbe-42f4-b9a9-a448567a00d7-kube-api-access-wdkxk\") pod \"payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh\" (UID: \"4c90ce47-9dbe-42f4-b9a9-a448567a00d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh"
Sep 08 16:48:50.728884 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.728861    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh"
Sep 08 16:48:50.963910 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:50.963823    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg"]
Sep 08 16:48:50.966546 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:48:50.966520    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e0f270f_0dcb_4261_baea_8a77e154f53b.slice/crio-3c4ee9dcbb5bb38e6b5b09e21dded109f37a490b54421a4b03c7cc01cac6fcd5 WatchSource:0}: Error finding container 3c4ee9dcbb5bb38e6b5b09e21dded109f37a490b54421a4b03c7cc01cac6fcd5: Status 404 returned error can't find the container with id 3c4ee9dcbb5bb38e6b5b09e21dded109f37a490b54421a4b03c7cc01cac6fcd5
Sep 08 16:48:51.012417 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:51.012391    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh"]
Sep 08 16:48:51.014619 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:48:51.014596    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c90ce47_9dbe_42f4_b9a9_a448567a00d7.slice/crio-a2efbb7c54f01e90c9e5cf135e116fd691e77562e9361d5006ec91a96c0370f6 WatchSource:0}: Error finding container a2efbb7c54f01e90c9e5cf135e116fd691e77562e9361d5006ec91a96c0370f6: Status 404 returned error can't find the container with id a2efbb7c54f01e90c9e5cf135e116fd691e77562e9361d5006ec91a96c0370f6
Sep 08 16:48:51.359386 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:51.359288    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg" event={"ID":"4e0f270f-0dcb-4261-baea-8a77e154f53b","Type":"ContainerStarted","Data":"1306b5c9dd1be4edb9d3289831eb80dac143fdb654ec3d6d83ecd533c0875d61"}
Sep 08 16:48:51.359386 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:51.359356    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg"
Sep 08 16:48:51.359386 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:51.359384    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg" event={"ID":"4e0f270f-0dcb-4261-baea-8a77e154f53b","Type":"ContainerStarted","Data":"3c4ee9dcbb5bb38e6b5b09e21dded109f37a490b54421a4b03c7cc01cac6fcd5"}
Sep 08 16:48:51.361019 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:51.360998    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh" event={"ID":"4c90ce47-9dbe-42f4-b9a9-a448567a00d7","Type":"ContainerStarted","Data":"363025dfcd6c2e2eaa8785969e8d78f9ed3ffd4be20220f7158546f7cf750185"}
Sep 08 16:48:51.361157 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:51.361034    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh" event={"ID":"4c90ce47-9dbe-42f4-b9a9-a448567a00d7","Type":"ContainerStarted","Data":"a2efbb7c54f01e90c9e5cf135e116fd691e77562e9361d5006ec91a96c0370f6"}
Sep 08 16:48:51.361157 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:51.361127    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh"
Sep 08 16:48:51.375333 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:51.375283    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg" podStartSLOduration=1.375266834 podStartE2EDuration="1.375266834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:48:51.374933628 +0000 UTC m=+1115.537905077" watchObservedRunningTime="2026-09-08 16:48:51.375266834 +0000 UTC m=+1115.538238282"
Sep 08 16:48:51.392075 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:51.392033    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh" podStartSLOduration=1.392016965 podStartE2EDuration="1.392016965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:48:51.391461314 +0000 UTC m=+1115.554432774" watchObservedRunningTime="2026-09-08 16:48:51.392016965 +0000 UTC m=+1115.554988416"
Sep 08 16:48:52.446113 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:52.446084    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d"]
Sep 08 16:48:52.449527 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:52.449511    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d"
Sep 08 16:48:52.451908 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:52.451877    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-1df9f8\""
Sep 08 16:48:52.451908 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:52.451877    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-1df9f8-dockercfg-r9mtf\""
Sep 08 16:48:52.460776 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:52.460755    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d"]
Sep 08 16:48:52.490996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:52.490973    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd"]
Sep 08 16:48:52.494395 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:52.494380    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd"
Sep 08 16:48:52.504403 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:52.504384    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd"]
Sep 08 16:48:52.616606 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:52.616570    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwvcf\" (UniqueName: \"kubernetes.io/projected/24250dbd-430b-46ea-bf70-0baa9072373e-kube-api-access-cwvcf\") pod \"payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d\" (UID: \"24250dbd-430b-46ea-bf70-0baa9072373e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d"
Sep 08 16:48:52.616606 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:52.616605    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqh8p\" (UniqueName: \"kubernetes.io/projected/4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e-kube-api-access-fqh8p\") pod \"payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd\" (UID: \"4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd"
Sep 08 16:48:52.616814 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:52.616679    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24250dbd-430b-46ea-bf70-0baa9072373e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d\" (UID: \"24250dbd-430b-46ea-bf70-0baa9072373e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d"
Sep 08 16:48:52.616814 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:52.616738    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd\" (UID: \"4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd"
Sep 08 16:48:52.722725 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:52.722650    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwvcf\" (UniqueName: \"kubernetes.io/projected/24250dbd-430b-46ea-bf70-0baa9072373e-kube-api-access-cwvcf\") pod \"payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d\" (UID: \"24250dbd-430b-46ea-bf70-0baa9072373e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d"
Sep 08 16:48:52.722898 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:52.722858    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqh8p\" (UniqueName: \"kubernetes.io/projected/4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e-kube-api-access-fqh8p\") pod \"payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd\" (UID: \"4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd"
Sep 08 16:48:52.722965 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:52.722933    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24250dbd-430b-46ea-bf70-0baa9072373e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d\" (UID: \"24250dbd-430b-46ea-bf70-0baa9072373e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d"
Sep 08 16:48:52.723043 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:52.723020    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd\" (UID: \"4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd"
Sep 08 16:48:52.723751 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:52.723730    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24250dbd-430b-46ea-bf70-0baa9072373e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d\" (UID: \"24250dbd-430b-46ea-bf70-0baa9072373e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d"
Sep 08 16:48:52.723793 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:52.723737    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd\" (UID: \"4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd"
Sep 08 16:48:52.730701 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:52.730680    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqh8p\" (UniqueName: \"kubernetes.io/projected/4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e-kube-api-access-fqh8p\") pod \"payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd\" (UID: \"4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd"
Sep 08 16:48:52.730953 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:52.730935    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwvcf\" (UniqueName: \"kubernetes.io/projected/24250dbd-430b-46ea-bf70-0baa9072373e-kube-api-access-cwvcf\") pod \"payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d\" (UID: \"24250dbd-430b-46ea-bf70-0baa9072373e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d"
Sep 08 16:48:52.760785 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:52.760763    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d"
Sep 08 16:48:52.803681 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:52.803653    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd"
Sep 08 16:48:53.034576 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:53.034388    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d"]
Sep 08 16:48:53.037427 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:48:53.037385    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24250dbd_430b_46ea_bf70_0baa9072373e.slice/crio-cfe9530492f42e149ff7457a52d85c1f3940234b726584d33b96d054cd3ef943 WatchSource:0}: Error finding container cfe9530492f42e149ff7457a52d85c1f3940234b726584d33b96d054cd3ef943: Status 404 returned error can't find the container with id cfe9530492f42e149ff7457a52d85c1f3940234b726584d33b96d054cd3ef943
Sep 08 16:48:53.085709 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:53.085687    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd"]
Sep 08 16:48:53.087663 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:48:53.087636    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ccf9b5f_ca47_49ee_a0a9_a6e58f15b17e.slice/crio-e64f9fe04569c34b49ad06f31f3a77246054f1577271264ce2c678ce57601400 WatchSource:0}: Error finding container e64f9fe04569c34b49ad06f31f3a77246054f1577271264ce2c678ce57601400: Status 404 returned error can't find the container with id e64f9fe04569c34b49ad06f31f3a77246054f1577271264ce2c678ce57601400
Sep 08 16:48:53.374548 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:53.374453    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd" event={"ID":"4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e","Type":"ContainerStarted","Data":"e0f77d02bf7ee553407d44bcdb8bb5bd7849347ae86404f331cee0cee7defe1f"}
Sep 08 16:48:53.374548 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:53.374524    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd" event={"ID":"4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e","Type":"ContainerStarted","Data":"e64f9fe04569c34b49ad06f31f3a77246054f1577271264ce2c678ce57601400"}
Sep 08 16:48:53.374759 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:53.374650    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd"
Sep 08 16:48:53.376289 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:53.376256    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d" event={"ID":"24250dbd-430b-46ea-bf70-0baa9072373e","Type":"ContainerStarted","Data":"9fc2283be5f7477b7a1ab12a92bd05e5f0ee77ee6e53e5396e8771e7ba6097c9"}
Sep 08 16:48:53.376421 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:53.376300    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d" event={"ID":"24250dbd-430b-46ea-bf70-0baa9072373e","Type":"ContainerStarted","Data":"cfe9530492f42e149ff7457a52d85c1f3940234b726584d33b96d054cd3ef943"}
Sep 08 16:48:53.376421 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:53.376327    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d"
Sep 08 16:48:53.390470 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:53.390424    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd" podStartSLOduration=1.390411332 podStartE2EDuration="1.390411332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:48:53.389102195 +0000 UTC m=+1117.552073655" watchObservedRunningTime="2026-09-08 16:48:53.390411332 +0000 UTC m=+1117.553382778"
Sep 08 16:48:53.404763 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:53.404722    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d" podStartSLOduration=1.404706961 podStartE2EDuration="1.404706961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:48:53.404175254 +0000 UTC m=+1117.567146700" watchObservedRunningTime="2026-09-08 16:48:53.404706961 +0000 UTC m=+1117.567678410"
Sep 08 16:48:54.537040 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:48:54.537010    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:49:02.367029 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:02.366997    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh"
Sep 08 16:49:02.367541 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:02.367182    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg"
Sep 08 16:49:04.385107 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:04.385079    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d"
Sep 08 16:49:04.385559 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:04.385160    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd"
Sep 08 16:49:19.490585 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:19.490549    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5"]
Sep 08 16:49:19.495465 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:19.495446    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5"
Sep 08 16:49:19.497721 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:19.497697    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-6b15ca\""
Sep 08 16:49:19.497849 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:19.497704    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-6b15ca-dockercfg-nm8sc\""
Sep 08 16:49:19.504080 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:19.504061    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5"]
Sep 08 16:49:19.521814 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:19.521786    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxcvd\" (UniqueName: \"kubernetes.io/projected/55b25b67-ec47-49db-8f49-4836d3a35d01-kube-api-access-fxcvd\") pod \"payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5\" (UID: \"55b25b67-ec47-49db-8f49-4836d3a35d01\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5"
Sep 08 16:49:19.521950 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:19.521906    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55b25b67-ec47-49db-8f49-4836d3a35d01-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5\" (UID: \"55b25b67-ec47-49db-8f49-4836d3a35d01\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5"
Sep 08 16:49:19.522244 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:19.522226    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk"]
Sep 08 16:49:19.525470 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:19.525455    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk"
Sep 08 16:49:19.536496 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:19.536475    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk"]
Sep 08 16:49:19.622710 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:19.622679    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55b25b67-ec47-49db-8f49-4836d3a35d01-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5\" (UID: \"55b25b67-ec47-49db-8f49-4836d3a35d01\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5"
Sep 08 16:49:19.622910 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:19.622733    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78146cde-eba7-4257-b601-479c516df11c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk\" (UID: \"78146cde-eba7-4257-b601-479c516df11c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk"
Sep 08 16:49:19.622910 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:19.622755    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxcvd\" (UniqueName: \"kubernetes.io/projected/55b25b67-ec47-49db-8f49-4836d3a35d01-kube-api-access-fxcvd\") pod \"payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5\" (UID: \"55b25b67-ec47-49db-8f49-4836d3a35d01\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5"
Sep 08 16:49:19.622910 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:19.622772    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfwhx\" (UniqueName: \"kubernetes.io/projected/78146cde-eba7-4257-b601-479c516df11c-kube-api-access-lfwhx\") pod \"payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk\" (UID: \"78146cde-eba7-4257-b601-479c516df11c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk"
Sep 08 16:49:19.623362 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:19.623344    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55b25b67-ec47-49db-8f49-4836d3a35d01-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5\" (UID: \"55b25b67-ec47-49db-8f49-4836d3a35d01\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5"
Sep 08 16:49:19.630761 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:19.630736    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxcvd\" (UniqueName: \"kubernetes.io/projected/55b25b67-ec47-49db-8f49-4836d3a35d01-kube-api-access-fxcvd\") pod \"payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5\" (UID: \"55b25b67-ec47-49db-8f49-4836d3a35d01\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5"
Sep 08 16:49:19.723193 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:19.723155    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78146cde-eba7-4257-b601-479c516df11c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk\" (UID: \"78146cde-eba7-4257-b601-479c516df11c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk"
Sep 08 16:49:19.723193 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:19.723196    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfwhx\" (UniqueName: \"kubernetes.io/projected/78146cde-eba7-4257-b601-479c516df11c-kube-api-access-lfwhx\") pod \"payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk\" (UID: \"78146cde-eba7-4257-b601-479c516df11c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk"
Sep 08 16:49:19.723865 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:19.723816    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78146cde-eba7-4257-b601-479c516df11c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk\" (UID: \"78146cde-eba7-4257-b601-479c516df11c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk"
Sep 08 16:49:19.731656 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:19.731626    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfwhx\" (UniqueName: \"kubernetes.io/projected/78146cde-eba7-4257-b601-479c516df11c-kube-api-access-lfwhx\") pod \"payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk\" (UID: \"78146cde-eba7-4257-b601-479c516df11c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk"
Sep 08 16:49:19.806304 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:19.806222    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5"
Sep 08 16:49:19.835368 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:19.835337    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk"
Sep 08 16:49:20.078763 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:20.078735    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5"]
Sep 08 16:49:20.081345 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:49:20.081308    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b25b67_ec47_49db_8f49_4836d3a35d01.slice/crio-c3b567c8eb0be064ca31b467d714e98d244ae4954789231bd81c40394d51ad56 WatchSource:0}: Error finding container c3b567c8eb0be064ca31b467d714e98d244ae4954789231bd81c40394d51ad56: Status 404 returned error can't find the container with id c3b567c8eb0be064ca31b467d714e98d244ae4954789231bd81c40394d51ad56
Sep 08 16:49:20.118049 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:20.117990    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk"]
Sep 08 16:49:20.120439 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:49:20.120415    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78146cde_eba7_4257_b601_479c516df11c.slice/crio-d0ed89c7dd9a1a71ad562a1aafc3ff57f746670f74ab1a9fd02426a352fc11dc WatchSource:0}: Error finding container d0ed89c7dd9a1a71ad562a1aafc3ff57f746670f74ab1a9fd02426a352fc11dc: Status 404 returned error can't find the container with id d0ed89c7dd9a1a71ad562a1aafc3ff57f746670f74ab1a9fd02426a352fc11dc
Sep 08 16:49:20.518910 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:20.518869    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk" event={"ID":"78146cde-eba7-4257-b601-479c516df11c","Type":"ContainerStarted","Data":"a4f4f9bcdae310f9965f9e9632bd8211f3910137e1f4dd41cd4033f462e85e58"}
Sep 08 16:49:20.519400 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:20.518939    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk"
Sep 08 16:49:20.519400 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:20.518963    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk" event={"ID":"78146cde-eba7-4257-b601-479c516df11c","Type":"ContainerStarted","Data":"d0ed89c7dd9a1a71ad562a1aafc3ff57f746670f74ab1a9fd02426a352fc11dc"}
Sep 08 16:49:20.520616 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:20.520584    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5" event={"ID":"55b25b67-ec47-49db-8f49-4836d3a35d01","Type":"ContainerStarted","Data":"2cacdfdd576bbfd7fe20d80c331f92a2d3c378a2e269dab429fbf8b9f0e3030b"}
Sep 08 16:49:20.520733 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:20.520631    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5" event={"ID":"55b25b67-ec47-49db-8f49-4836d3a35d01","Type":"ContainerStarted","Data":"c3b567c8eb0be064ca31b467d714e98d244ae4954789231bd81c40394d51ad56"}
Sep 08 16:49:20.520733 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:20.520652    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5"
Sep 08 16:49:20.535792 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:20.535748    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk" podStartSLOduration=1.535733778 podStartE2EDuration="1.535733778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:49:20.534200606 +0000 UTC m=+1144.697172067" watchObservedRunningTime="2026-09-08 16:49:20.535733778 +0000 UTC m=+1144.698705221"
Sep 08 16:49:20.550914 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:20.550874    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5" podStartSLOduration=1.550859928 podStartE2EDuration="1.550859928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:49:20.550402878 +0000 UTC m=+1144.713374337" watchObservedRunningTime="2026-09-08 16:49:20.550859928 +0000 UTC m=+1144.713831375"
Sep 08 16:49:23.544733 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:23.544699    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:49:24.619521 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:24.619484    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk"]
Sep 08 16:49:24.624124 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:24.624103    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk"
Sep 08 16:49:24.626736 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:24.626715    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-6e5401-dockercfg-q7rkv\""
Sep 08 16:49:24.626908 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:24.626746    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-6e5401\""
Sep 08 16:49:24.629450 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:24.629393    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk"]
Sep 08 16:49:24.633140 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:24.633120    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk"
Sep 08 16:49:24.633584 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:24.633565    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk"]
Sep 08 16:49:24.643588 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:24.643565    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk"]
Sep 08 16:49:24.668293 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:24.668265    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71fadf7d-46e1-4a4a-abc9-8276185c31e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk\" (UID: \"71fadf7d-46e1-4a4a-abc9-8276185c31e6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk"
Sep 08 16:49:24.668403 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:24.668310    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxnpj\" (UniqueName: \"kubernetes.io/projected/71fadf7d-46e1-4a4a-abc9-8276185c31e6-kube-api-access-sxnpj\") pod \"payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk\" (UID: \"71fadf7d-46e1-4a4a-abc9-8276185c31e6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk"
Sep 08 16:49:24.668403 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:24.668372    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z77x\" (UniqueName: \"kubernetes.io/projected/84aaa926-730e-460d-bba8-36bfff099744-kube-api-access-2z77x\") pod \"payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk\" (UID: \"84aaa926-730e-460d-bba8-36bfff099744\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk"
Sep 08 16:49:24.668481 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:24.668421    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84aaa926-730e-460d-bba8-36bfff099744-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk\" (UID: \"84aaa926-730e-460d-bba8-36bfff099744\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk"
Sep 08 16:49:24.769322 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:24.769291    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71fadf7d-46e1-4a4a-abc9-8276185c31e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk\" (UID: \"71fadf7d-46e1-4a4a-abc9-8276185c31e6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk"
Sep 08 16:49:24.769491 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:24.769337    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxnpj\" (UniqueName: \"kubernetes.io/projected/71fadf7d-46e1-4a4a-abc9-8276185c31e6-kube-api-access-sxnpj\") pod \"payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk\" (UID: \"71fadf7d-46e1-4a4a-abc9-8276185c31e6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk"
Sep 08 16:49:24.769491 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:24.769383    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z77x\" (UniqueName: \"kubernetes.io/projected/84aaa926-730e-460d-bba8-36bfff099744-kube-api-access-2z77x\") pod \"payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk\" (UID: \"84aaa926-730e-460d-bba8-36bfff099744\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk"
Sep 08 16:49:24.769491 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:24.769422    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84aaa926-730e-460d-bba8-36bfff099744-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk\" (UID: \"84aaa926-730e-460d-bba8-36bfff099744\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk"
Sep 08 16:49:24.770089 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:24.770069    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84aaa926-730e-460d-bba8-36bfff099744-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk\" (UID: \"84aaa926-730e-460d-bba8-36bfff099744\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk"
Sep 08 16:49:24.770180 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:24.770078    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71fadf7d-46e1-4a4a-abc9-8276185c31e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk\" (UID: \"71fadf7d-46e1-4a4a-abc9-8276185c31e6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk"
Sep 08 16:49:24.778501 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:24.778468    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxnpj\" (UniqueName: \"kubernetes.io/projected/71fadf7d-46e1-4a4a-abc9-8276185c31e6-kube-api-access-sxnpj\") pod \"payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk\" (UID: \"71fadf7d-46e1-4a4a-abc9-8276185c31e6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk"
Sep 08 16:49:24.778618 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:24.778473    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z77x\" (UniqueName: \"kubernetes.io/projected/84aaa926-730e-460d-bba8-36bfff099744-kube-api-access-2z77x\") pod \"payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk\" (UID: \"84aaa926-730e-460d-bba8-36bfff099744\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk"
Sep 08 16:49:24.936014 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:24.935985    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk"
Sep 08 16:49:24.945806 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:24.945753    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk"
Sep 08 16:49:25.211708 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:25.211581    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk"]
Sep 08 16:49:25.214002 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:49:25.213974    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71fadf7d_46e1_4a4a_abc9_8276185c31e6.slice/crio-4865394dfb6d6ef61512c8d49e150b7a5dbc001d49bfd141da7b63e77f854236 WatchSource:0}: Error finding container 4865394dfb6d6ef61512c8d49e150b7a5dbc001d49bfd141da7b63e77f854236: Status 404 returned error can't find the container with id 4865394dfb6d6ef61512c8d49e150b7a5dbc001d49bfd141da7b63e77f854236
Sep 08 16:49:25.228799 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:25.228773    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk"]
Sep 08 16:49:25.230048 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:49:25.230025    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84aaa926_730e_460d_bba8_36bfff099744.slice/crio-805cae092cadb869d102b3c50c3db68632dd440a35217cd4d0cbd366c637f65d WatchSource:0}: Error finding container 805cae092cadb869d102b3c50c3db68632dd440a35217cd4d0cbd366c637f65d: Status 404 returned error can't find the container with id 805cae092cadb869d102b3c50c3db68632dd440a35217cd4d0cbd366c637f65d
Sep 08 16:49:25.551267 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:25.551160    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk" event={"ID":"84aaa926-730e-460d-bba8-36bfff099744","Type":"ContainerStarted","Data":"125da4abeaca6239f6a66cbbaa260c1cafeccaee185b7b24c5764434f2586760"}
Sep 08 16:49:25.551267 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:25.551232    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk"
Sep 08 16:49:25.551267 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:25.551257    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk" event={"ID":"84aaa926-730e-460d-bba8-36bfff099744","Type":"ContainerStarted","Data":"805cae092cadb869d102b3c50c3db68632dd440a35217cd4d0cbd366c637f65d"}
Sep 08 16:49:25.552761 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:25.552738    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk" event={"ID":"71fadf7d-46e1-4a4a-abc9-8276185c31e6","Type":"ContainerStarted","Data":"4815347b806004b749d231e6078cc158b72af08ee93b20e2da8e643d2cd603a5"}
Sep 08 16:49:25.552917 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:25.552787    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk" event={"ID":"71fadf7d-46e1-4a4a-abc9-8276185c31e6","Type":"ContainerStarted","Data":"4865394dfb6d6ef61512c8d49e150b7a5dbc001d49bfd141da7b63e77f854236"}
Sep 08 16:49:25.552917 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:25.552820    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk"
Sep 08 16:49:25.567546 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:25.567502    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk" podStartSLOduration=1.5674837670000001 podStartE2EDuration="1.567483767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:49:25.566574068 +0000 UTC m=+1149.729545516" watchObservedRunningTime="2026-09-08 16:49:25.567483767 +0000 UTC m=+1149.730455220"
Sep 08 16:49:25.584286 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:25.584232    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk" podStartSLOduration=1.584213748 podStartE2EDuration="1.584213748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:49:25.583199929 +0000 UTC m=+1149.746171379" watchObservedRunningTime="2026-09-08 16:49:25.584213748 +0000 UTC m=+1149.747185197"
Sep 08 16:49:31.529246 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:31.529210    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk"
Sep 08 16:49:31.529694 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:31.529291    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5"
Sep 08 16:49:33.327460 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:33.327419    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:49:36.559752 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:36.559724    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk"
Sep 08 16:49:36.560234 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:36.559785    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk"
Sep 08 16:49:47.428867 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.428809    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"]
Sep 08 16:49:47.433239 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.433214    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:49:47.436588 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.436560    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-608268\"/\"default-dockercfg-nj6gs\""
Sep 08 16:49:47.436760 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.436561    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-608268\"/\"openshift-service-ca.crt\""
Sep 08 16:49:47.436760 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.436600    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-608268\"/\"kube-root-ca.crt\""
Sep 08 16:49:47.436760 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.436561    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-608268\"/\"auth-test-model-608268-kserve-self-signed-certs\""
Sep 08 16:49:47.443077 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.443051    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"]
Sep 08 16:49:47.574043 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.573993    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55zm9\" (UniqueName: \"kubernetes.io/projected/37059f8f-c983-4786-8836-2f07d17dc8cf-kube-api-access-55zm9\") pod \"auth-test-model-608268-kserve-7586d5f75f-8d6vk\" (UID: \"37059f8f-c983-4786-8836-2f07d17dc8cf\") " pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:49:47.574241 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.574065    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37059f8f-c983-4786-8836-2f07d17dc8cf-tls-certs\") pod \"auth-test-model-608268-kserve-7586d5f75f-8d6vk\" (UID: \"37059f8f-c983-4786-8836-2f07d17dc8cf\") " pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:49:47.574241 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.574141    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37059f8f-c983-4786-8836-2f07d17dc8cf-home\") pod \"auth-test-model-608268-kserve-7586d5f75f-8d6vk\" (UID: \"37059f8f-c983-4786-8836-2f07d17dc8cf\") " pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:49:47.574241 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.574165    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37059f8f-c983-4786-8836-2f07d17dc8cf-model-cache\") pod \"auth-test-model-608268-kserve-7586d5f75f-8d6vk\" (UID: \"37059f8f-c983-4786-8836-2f07d17dc8cf\") " pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:49:47.574241 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.574190    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37059f8f-c983-4786-8836-2f07d17dc8cf-tmp-dir\") pod \"auth-test-model-608268-kserve-7586d5f75f-8d6vk\" (UID: \"37059f8f-c983-4786-8836-2f07d17dc8cf\") " pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:49:47.574451 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.574328    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37059f8f-c983-4786-8836-2f07d17dc8cf-dshm\") pod \"auth-test-model-608268-kserve-7586d5f75f-8d6vk\" (UID: \"37059f8f-c983-4786-8836-2f07d17dc8cf\") " pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:49:47.675030 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.674991    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37059f8f-c983-4786-8836-2f07d17dc8cf-home\") pod \"auth-test-model-608268-kserve-7586d5f75f-8d6vk\" (UID: \"37059f8f-c983-4786-8836-2f07d17dc8cf\") " pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:49:47.675234 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.675045    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37059f8f-c983-4786-8836-2f07d17dc8cf-model-cache\") pod \"auth-test-model-608268-kserve-7586d5f75f-8d6vk\" (UID: \"37059f8f-c983-4786-8836-2f07d17dc8cf\") " pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:49:47.675234 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.675070    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37059f8f-c983-4786-8836-2f07d17dc8cf-tmp-dir\") pod \"auth-test-model-608268-kserve-7586d5f75f-8d6vk\" (UID: \"37059f8f-c983-4786-8836-2f07d17dc8cf\") " pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:49:47.675234 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.675142    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37059f8f-c983-4786-8836-2f07d17dc8cf-dshm\") pod \"auth-test-model-608268-kserve-7586d5f75f-8d6vk\" (UID: \"37059f8f-c983-4786-8836-2f07d17dc8cf\") " pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:49:47.675234 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.675175    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55zm9\" (UniqueName: \"kubernetes.io/projected/37059f8f-c983-4786-8836-2f07d17dc8cf-kube-api-access-55zm9\") pod \"auth-test-model-608268-kserve-7586d5f75f-8d6vk\" (UID: \"37059f8f-c983-4786-8836-2f07d17dc8cf\") " pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:49:47.675234 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.675217    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37059f8f-c983-4786-8836-2f07d17dc8cf-tls-certs\") pod \"auth-test-model-608268-kserve-7586d5f75f-8d6vk\" (UID: \"37059f8f-c983-4786-8836-2f07d17dc8cf\") " pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:49:47.675512 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.675427    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37059f8f-c983-4786-8836-2f07d17dc8cf-home\") pod \"auth-test-model-608268-kserve-7586d5f75f-8d6vk\" (UID: \"37059f8f-c983-4786-8836-2f07d17dc8cf\") " pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:49:47.675773 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.675752    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37059f8f-c983-4786-8836-2f07d17dc8cf-tmp-dir\") pod \"auth-test-model-608268-kserve-7586d5f75f-8d6vk\" (UID: \"37059f8f-c983-4786-8836-2f07d17dc8cf\") " pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:49:47.675978 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.675955    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37059f8f-c983-4786-8836-2f07d17dc8cf-model-cache\") pod \"auth-test-model-608268-kserve-7586d5f75f-8d6vk\" (UID: \"37059f8f-c983-4786-8836-2f07d17dc8cf\") " pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:49:47.677904 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.677874    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37059f8f-c983-4786-8836-2f07d17dc8cf-dshm\") pod \"auth-test-model-608268-kserve-7586d5f75f-8d6vk\" (UID: \"37059f8f-c983-4786-8836-2f07d17dc8cf\") " pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:49:47.678178 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.678161    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37059f8f-c983-4786-8836-2f07d17dc8cf-tls-certs\") pod \"auth-test-model-608268-kserve-7586d5f75f-8d6vk\" (UID: \"37059f8f-c983-4786-8836-2f07d17dc8cf\") " pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:49:47.683978 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.683920    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55zm9\" (UniqueName: \"kubernetes.io/projected/37059f8f-c983-4786-8836-2f07d17dc8cf-kube-api-access-55zm9\") pod \"auth-test-model-608268-kserve-7586d5f75f-8d6vk\" (UID: \"37059f8f-c983-4786-8836-2f07d17dc8cf\") " pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:49:47.744952 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:47.744905    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:49:48.023582 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:48.023493    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"]
Sep 08 16:49:48.026308 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:49:48.026276    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37059f8f_c983_4786_8836_2f07d17dc8cf.slice/crio-22a5a52244621cdaba337bbbe06ace7171801901513d904850ff7403d508aa17 WatchSource:0}: Error finding container 22a5a52244621cdaba337bbbe06ace7171801901513d904850ff7403d508aa17: Status 404 returned error can't find the container with id 22a5a52244621cdaba337bbbe06ace7171801901513d904850ff7403d508aa17
Sep 08 16:49:48.683127 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:48.683087    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk" event={"ID":"37059f8f-c983-4786-8836-2f07d17dc8cf","Type":"ContainerStarted","Data":"6f5d22ec9721e02584a9629b22040d5cb941aceeb2d84d0c5074e554485a19cb"}
Sep 08 16:49:48.683583 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:48.683142    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk" event={"ID":"37059f8f-c983-4786-8836-2f07d17dc8cf","Type":"ContainerStarted","Data":"22a5a52244621cdaba337bbbe06ace7171801901513d904850ff7403d508aa17"}
Sep 08 16:49:48.701178 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:48.701118    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk" podStartSLOduration=1.7011006929999999 podStartE2EDuration="1.701100693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:49:48.700423286 +0000 UTC m=+1172.863394725" watchObservedRunningTime="2026-09-08 16:49:48.701100693 +0000 UTC m=+1172.864072141"
Sep 08 16:49:51.729324 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:51.729283    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:49:57.745652 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:57.745615    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:49:57.746072 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:57.745667    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:49:57.758597 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:57.758580    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:49:58.749552 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:49:58.749526    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:50:06.714212 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:06.714173    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr"]
Sep 08 16:50:06.719093 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:06.719076    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr"
Sep 08 16:50:06.721704 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:06.721683    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e43c7719-dockercfg-mkwkw\""
Sep 08 16:50:06.721849 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:06.721750    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e43c7719\""
Sep 08 16:50:06.728643 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:06.728616    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr"]
Sep 08 16:50:06.735015 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:06.734987    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b03022d4-b768-4dc9-b20d-4bed68b0012e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr\" (UID: \"b03022d4-b768-4dc9-b20d-4bed68b0012e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr"
Sep 08 16:50:06.735151 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:06.735028    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqjd8\" (UniqueName: \"kubernetes.io/projected/b03022d4-b768-4dc9-b20d-4bed68b0012e-kube-api-access-gqjd8\") pod \"payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr\" (UID: \"b03022d4-b768-4dc9-b20d-4bed68b0012e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr"
Sep 08 16:50:06.743150 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:06.743124    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf"]
Sep 08 16:50:06.746673 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:06.746656    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf"
Sep 08 16:50:06.762301 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:06.762274    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf"]
Sep 08 16:50:06.836256 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:06.836216    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqjd8\" (UniqueName: \"kubernetes.io/projected/b03022d4-b768-4dc9-b20d-4bed68b0012e-kube-api-access-gqjd8\") pod \"payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr\" (UID: \"b03022d4-b768-4dc9-b20d-4bed68b0012e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr"
Sep 08 16:50:06.836456 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:06.836314    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rl79\" (UniqueName: \"kubernetes.io/projected/38a2b5c4-76b0-42ac-9803-18fcc1989abe-kube-api-access-6rl79\") pod \"payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf\" (UID: \"38a2b5c4-76b0-42ac-9803-18fcc1989abe\") " pod="openshift-ingress/payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf"
Sep 08 16:50:06.836456 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:06.836430    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b03022d4-b768-4dc9-b20d-4bed68b0012e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr\" (UID: \"b03022d4-b768-4dc9-b20d-4bed68b0012e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr"
Sep 08 16:50:06.836577 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:06.836466    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a2b5c4-76b0-42ac-9803-18fcc1989abe-plugins-config-volume\") pod \"payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf\" (UID: \"38a2b5c4-76b0-42ac-9803-18fcc1989abe\") " pod="openshift-ingress/payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf"
Sep 08 16:50:06.837761 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:06.837729    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b03022d4-b768-4dc9-b20d-4bed68b0012e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr\" (UID: \"b03022d4-b768-4dc9-b20d-4bed68b0012e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr"
Sep 08 16:50:06.849614 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:06.849578    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqjd8\" (UniqueName: \"kubernetes.io/projected/b03022d4-b768-4dc9-b20d-4bed68b0012e-kube-api-access-gqjd8\") pod \"payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr\" (UID: \"b03022d4-b768-4dc9-b20d-4bed68b0012e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr"
Sep 08 16:50:06.937805 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:06.937767    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rl79\" (UniqueName: \"kubernetes.io/projected/38a2b5c4-76b0-42ac-9803-18fcc1989abe-kube-api-access-6rl79\") pod \"payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf\" (UID: \"38a2b5c4-76b0-42ac-9803-18fcc1989abe\") " pod="openshift-ingress/payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf"
Sep 08 16:50:06.937992 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:06.937879    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a2b5c4-76b0-42ac-9803-18fcc1989abe-plugins-config-volume\") pod \"payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf\" (UID: \"38a2b5c4-76b0-42ac-9803-18fcc1989abe\") " pod="openshift-ingress/payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf"
Sep 08 16:50:06.938531 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:06.938512    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a2b5c4-76b0-42ac-9803-18fcc1989abe-plugins-config-volume\") pod \"payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf\" (UID: \"38a2b5c4-76b0-42ac-9803-18fcc1989abe\") " pod="openshift-ingress/payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf"
Sep 08 16:50:06.954789 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:06.954764    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rl79\" (UniqueName: \"kubernetes.io/projected/38a2b5c4-76b0-42ac-9803-18fcc1989abe-kube-api-access-6rl79\") pod \"payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf\" (UID: \"38a2b5c4-76b0-42ac-9803-18fcc1989abe\") " pod="openshift-ingress/payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf"
Sep 08 16:50:07.031104 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:07.031007    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr"
Sep 08 16:50:07.056958 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:07.056929    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf"
Sep 08 16:50:07.306253 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:07.306224    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr"]
Sep 08 16:50:07.308429 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:50:07.308402    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03022d4_b768_4dc9_b20d_4bed68b0012e.slice/crio-39356a9a8502176baf9e82a85db4beecaaad5ff59f0b3a6e3e6bfea0bae930da WatchSource:0}: Error finding container 39356a9a8502176baf9e82a85db4beecaaad5ff59f0b3a6e3e6bfea0bae930da: Status 404 returned error can't find the container with id 39356a9a8502176baf9e82a85db4beecaaad5ff59f0b3a6e3e6bfea0bae930da
Sep 08 16:50:07.339060 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:07.339038    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf"]
Sep 08 16:50:07.343929 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:50:07.343904    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a2b5c4_76b0_42ac_9803_18fcc1989abe.slice/crio-b9e55e43811abf308db95a36b524fd6fc630f78ce9ba896c8c750235313feb74 WatchSource:0}: Error finding container b9e55e43811abf308db95a36b524fd6fc630f78ce9ba896c8c750235313feb74: Status 404 returned error can't find the container with id b9e55e43811abf308db95a36b524fd6fc630f78ce9ba896c8c750235313feb74
Sep 08 16:50:07.790317 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:07.790266    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf" event={"ID":"38a2b5c4-76b0-42ac-9803-18fcc1989abe","Type":"ContainerStarted","Data":"b5e18714679757201c1b36cd034b3cc70961571e6031547757f316b44f39adf2"}
Sep 08 16:50:07.790788 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:07.790335    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf" event={"ID":"38a2b5c4-76b0-42ac-9803-18fcc1989abe","Type":"ContainerStarted","Data":"b9e55e43811abf308db95a36b524fd6fc630f78ce9ba896c8c750235313feb74"}
Sep 08 16:50:07.790788 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:07.790494    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf"
Sep 08 16:50:07.791979 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:07.791957    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr" event={"ID":"b03022d4-b768-4dc9-b20d-4bed68b0012e","Type":"ContainerStarted","Data":"782418eb1702c0b623ee427c9a36da471274e266ba365721f614b7fb418d4fa6"}
Sep 08 16:50:07.792154 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:07.791999    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr" event={"ID":"b03022d4-b768-4dc9-b20d-4bed68b0012e","Type":"ContainerStarted","Data":"39356a9a8502176baf9e82a85db4beecaaad5ff59f0b3a6e3e6bfea0bae930da"}
Sep 08 16:50:07.792154 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:07.792055    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr"
Sep 08 16:50:07.807368 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:07.807315    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf" podStartSLOduration=1.8072963039999999 podStartE2EDuration="1.807296304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:50:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:50:07.806017555 +0000 UTC m=+1191.968989004" watchObservedRunningTime="2026-09-08 16:50:07.807296304 +0000 UTC m=+1191.970267757"
Sep 08 16:50:07.821883 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:07.821818    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr" podStartSLOduration=1.821802983 podStartE2EDuration="1.821802983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:50:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:50:07.821233229 +0000 UTC m=+1191.984204680" watchObservedRunningTime="2026-09-08 16:50:07.821802983 +0000 UTC m=+1191.984774827"
Sep 08 16:50:08.109879 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.109786    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"]
Sep 08 16:50:08.113597 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.113577    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:50:08.116513 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.116492    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6b15ca\"/\"kube-root-ca.crt\""
Sep 08 16:50:08.116649 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.116514    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6b15ca\"/\"openshift-service-ca.crt\""
Sep 08 16:50:08.116649 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.116540    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6b15ca\"/\"auth-test-model-6b15ca-kserve-self-signed-certs\""
Sep 08 16:50:08.117020 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.117004    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6b15ca\"/\"default-dockercfg-c8vzm\""
Sep 08 16:50:08.124044 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.124022    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"]
Sep 08 16:50:08.147672 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.147645    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54965c1c-2b57-4ec0-bbee-9115db00bb0b-model-cache\") pod \"auth-test-model-6b15ca-kserve-587578f459-mws74\" (UID: \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:50:08.147759 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.147682    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54965c1c-2b57-4ec0-bbee-9115db00bb0b-tmp-dir\") pod \"auth-test-model-6b15ca-kserve-587578f459-mws74\" (UID: \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:50:08.147759 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.147712    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk66r\" (UniqueName: \"kubernetes.io/projected/54965c1c-2b57-4ec0-bbee-9115db00bb0b-kube-api-access-wk66r\") pod \"auth-test-model-6b15ca-kserve-587578f459-mws74\" (UID: \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:50:08.147860 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.147785    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54965c1c-2b57-4ec0-bbee-9115db00bb0b-home\") pod \"auth-test-model-6b15ca-kserve-587578f459-mws74\" (UID: \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:50:08.147903 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.147825    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54965c1c-2b57-4ec0-bbee-9115db00bb0b-tls-certs\") pod \"auth-test-model-6b15ca-kserve-587578f459-mws74\" (UID: \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:50:08.147938 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.147921    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54965c1c-2b57-4ec0-bbee-9115db00bb0b-dshm\") pod \"auth-test-model-6b15ca-kserve-587578f459-mws74\" (UID: \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:50:08.249196 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.249163    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54965c1c-2b57-4ec0-bbee-9115db00bb0b-model-cache\") pod \"auth-test-model-6b15ca-kserve-587578f459-mws74\" (UID: \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:50:08.249355 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.249209    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54965c1c-2b57-4ec0-bbee-9115db00bb0b-tmp-dir\") pod \"auth-test-model-6b15ca-kserve-587578f459-mws74\" (UID: \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:50:08.249355 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.249232    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk66r\" (UniqueName: \"kubernetes.io/projected/54965c1c-2b57-4ec0-bbee-9115db00bb0b-kube-api-access-wk66r\") pod \"auth-test-model-6b15ca-kserve-587578f459-mws74\" (UID: \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:50:08.249355 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.249257    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54965c1c-2b57-4ec0-bbee-9115db00bb0b-home\") pod \"auth-test-model-6b15ca-kserve-587578f459-mws74\" (UID: \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:50:08.249355 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.249281    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54965c1c-2b57-4ec0-bbee-9115db00bb0b-tls-certs\") pod \"auth-test-model-6b15ca-kserve-587578f459-mws74\" (UID: \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:50:08.249355 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.249320    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54965c1c-2b57-4ec0-bbee-9115db00bb0b-dshm\") pod \"auth-test-model-6b15ca-kserve-587578f459-mws74\" (UID: \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:50:08.249611 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.249583    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54965c1c-2b57-4ec0-bbee-9115db00bb0b-model-cache\") pod \"auth-test-model-6b15ca-kserve-587578f459-mws74\" (UID: \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:50:08.249611 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.249601    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54965c1c-2b57-4ec0-bbee-9115db00bb0b-tmp-dir\") pod \"auth-test-model-6b15ca-kserve-587578f459-mws74\" (UID: \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:50:08.249705 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.249650    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54965c1c-2b57-4ec0-bbee-9115db00bb0b-home\") pod \"auth-test-model-6b15ca-kserve-587578f459-mws74\" (UID: \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:50:08.251777 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.251753    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54965c1c-2b57-4ec0-bbee-9115db00bb0b-dshm\") pod \"auth-test-model-6b15ca-kserve-587578f459-mws74\" (UID: \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:50:08.252026 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.252007    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54965c1c-2b57-4ec0-bbee-9115db00bb0b-tls-certs\") pod \"auth-test-model-6b15ca-kserve-587578f459-mws74\" (UID: \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:50:08.257575 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.257557    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk66r\" (UniqueName: \"kubernetes.io/projected/54965c1c-2b57-4ec0-bbee-9115db00bb0b-kube-api-access-wk66r\") pod \"auth-test-model-6b15ca-kserve-587578f459-mws74\" (UID: \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:50:08.424501 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.424463    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:50:08.709866 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.709814    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"]
Sep 08 16:50:08.712067 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:50:08.712043    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54965c1c_2b57_4ec0_bbee_9115db00bb0b.slice/crio-edbd4f57f2cb5b72bed04b2e3c6e2075384dd47af2e0ed3e7a69d2ea3897f985 WatchSource:0}: Error finding container edbd4f57f2cb5b72bed04b2e3c6e2075384dd47af2e0ed3e7a69d2ea3897f985: Status 404 returned error can't find the container with id edbd4f57f2cb5b72bed04b2e3c6e2075384dd47af2e0ed3e7a69d2ea3897f985
Sep 08 16:50:08.800092 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.800057    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74" event={"ID":"54965c1c-2b57-4ec0-bbee-9115db00bb0b","Type":"ContainerStarted","Data":"435a59f93b6bdd9bbb89ccf0ecb83c256f9e95b95ed648dfb2f0fe2a71906588"}
Sep 08 16:50:08.800554 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.800108    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74" event={"ID":"54965c1c-2b57-4ec0-bbee-9115db00bb0b","Type":"ContainerStarted","Data":"edbd4f57f2cb5b72bed04b2e3c6e2075384dd47af2e0ed3e7a69d2ea3897f985"}
Sep 08 16:50:08.816496 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:08.816444    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74" podStartSLOduration=0.816426093 podStartE2EDuration="816.426093ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:50:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:50:08.816264498 +0000 UTC m=+1192.979235945" watchObservedRunningTime="2026-09-08 16:50:08.816426093 +0000 UTC m=+1192.979397587"
Sep 08 16:50:14.429802 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:14.429766    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:50:16.440684 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:16.440646    2557 scope.go:117] "RemoveContainer" containerID="3b9b792bef2117e137aa65cb96448243551d2a0cc35154022919b4a74b7a6f27"
Sep 08 16:50:16.449872 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:16.449850    2557 scope.go:117] "RemoveContainer" containerID="d5f7587033f949373cd61b8510343b78b615919d377d9535116485d624da2c1d"
Sep 08 16:50:18.424679 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:18.424641    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:50:18.425121 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:18.424696    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:50:18.437526 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:18.437496    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:50:18.802796 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:18.802713    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf"
Sep 08 16:50:18.802796 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:18.802791    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr"
Sep 08 16:50:18.876617 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:18.876575    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:50:39.836558 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:39.836524    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:50:44.427318 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.427283    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"]
Sep 08 16:50:44.430824 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.430802    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:50:44.435629 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.435577    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e43c7719\"/\"e2e-lifecycle-model-e43c7719-kserve-self-signed-certs\""
Sep 08 16:50:44.435755 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.435633    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e43c7719\"/\"openshift-service-ca.crt\""
Sep 08 16:50:44.435755 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.435652    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e43c7719\"/\"default-dockercfg-kfqhs\""
Sep 08 16:50:44.435755 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.435738    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e43c7719\"/\"kube-root-ca.crt\""
Sep 08 16:50:44.441050 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.441026    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"]
Sep 08 16:50:44.480979 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.480954    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-model-cache\") pod \"e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w\" (UID: \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\") " pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:50:44.481112 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.480991    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-home\") pod \"e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w\" (UID: \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\") " pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:50:44.481112 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.481013    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-tls-certs\") pod \"e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w\" (UID: \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\") " pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:50:44.481112 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.481103    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-dshm\") pod \"e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w\" (UID: \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\") " pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:50:44.481289 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.481125    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-tmp-dir\") pod \"e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w\" (UID: \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\") " pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:50:44.481289 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.481186    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hxgf\" (UniqueName: \"kubernetes.io/projected/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-kube-api-access-6hxgf\") pod \"e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w\" (UID: \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\") " pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:50:44.582084 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.582054    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-dshm\") pod \"e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w\" (UID: \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\") " pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:50:44.582221 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.582093    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-tmp-dir\") pod \"e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w\" (UID: \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\") " pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:50:44.582221 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.582141    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hxgf\" (UniqueName: \"kubernetes.io/projected/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-kube-api-access-6hxgf\") pod \"e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w\" (UID: \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\") " pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:50:44.582221 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.582208    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-model-cache\") pod \"e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w\" (UID: \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\") " pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:50:44.582363 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.582245    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-home\") pod \"e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w\" (UID: \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\") " pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:50:44.582363 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.582277    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-tls-certs\") pod \"e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w\" (UID: \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\") " pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:50:44.582596 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.582567    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-tmp-dir\") pod \"e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w\" (UID: \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\") " pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:50:44.582729 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.582578    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-home\") pod \"e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w\" (UID: \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\") " pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:50:44.582729 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.582622    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-model-cache\") pod \"e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w\" (UID: \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\") " pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:50:44.584402 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.584371    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-dshm\") pod \"e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w\" (UID: \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\") " pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:50:44.584629 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.584614    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-tls-certs\") pod \"e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w\" (UID: \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\") " pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:50:44.590266 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.590244    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hxgf\" (UniqueName: \"kubernetes.io/projected/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-kube-api-access-6hxgf\") pod \"e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w\" (UID: \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\") " pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:50:44.740901 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:44.740812    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:50:45.015292 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:45.015270    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"]
Sep 08 16:50:45.017111 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:50:45.017086    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbdaa4b6_303a_4dd5_a0f7_be074d0c51bd.slice/crio-49017e7dc5d1a8a4e32f554f3262b421342f874cb955694664ee8a52671745eb WatchSource:0}: Error finding container 49017e7dc5d1a8a4e32f554f3262b421342f874cb955694664ee8a52671745eb: Status 404 returned error can't find the container with id 49017e7dc5d1a8a4e32f554f3262b421342f874cb955694664ee8a52671745eb
Sep 08 16:50:46.012902 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:46.012862    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w" event={"ID":"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd","Type":"ContainerStarted","Data":"ccaa5be2e1dfc0bd5da11bba06b8829609907d2202700a5bc472499032058788"}
Sep 08 16:50:46.013325 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:46.012911    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w" event={"ID":"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd","Type":"ContainerStarted","Data":"49017e7dc5d1a8a4e32f554f3262b421342f874cb955694664ee8a52671745eb"}
Sep 08 16:50:46.030313 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:46.030267    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w" podStartSLOduration=2.030251816 podStartE2EDuration="2.030251816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:50:46.028145596 +0000 UTC m=+1230.191117043" watchObservedRunningTime="2026-09-08 16:50:46.030251816 +0000 UTC m=+1230.193223263"
Sep 08 16:50:54.326914 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:54.326876    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:50:54.741169 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:54.741138    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:50:54.741347 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:54.741196    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:50:54.753612 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:54.753593    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:50:55.074506 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:50:55.074403    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:51:09.418215 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:09.418143    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e43c7719-786tx"]
Sep 08 16:51:09.421439 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:09.421422    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e43c7719-786tx"
Sep 08 16:51:09.423699 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:09.423678    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:51:09.423814 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:09.423783    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-886jf\""
Sep 08 16:51:09.428989 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:09.428967    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e43c7719-786tx"]
Sep 08 16:51:09.487247 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:09.487210    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ede2fd-bb4a-431a-a29b-f574cc6f421a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e43c7719-786tx\" (UID: \"06ede2fd-bb4a-431a-a29b-f574cc6f421a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e43c7719-786tx"
Sep 08 16:51:09.588261 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:09.588231    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ede2fd-bb4a-431a-a29b-f574cc6f421a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e43c7719-786tx\" (UID: \"06ede2fd-bb4a-431a-a29b-f574cc6f421a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e43c7719-786tx"
Sep 08 16:51:09.588845 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:09.588811    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ede2fd-bb4a-431a-a29b-f574cc6f421a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e43c7719-786tx\" (UID: \"06ede2fd-bb4a-431a-a29b-f574cc6f421a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e43c7719-786tx"
Sep 08 16:51:09.733473 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:09.733396    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e43c7719-786tx"
Sep 08 16:51:10.000857 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:10.000809    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e43c7719-786tx"]
Sep 08 16:51:10.003095 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:51:10.003064    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ede2fd_bb4a_431a_a29b_f574cc6f421a.slice/crio-0fe3a687ff5cd38df1043ae82936deec15eb39486f4bef821768d485ce17bf43 WatchSource:0}: Error finding container 0fe3a687ff5cd38df1043ae82936deec15eb39486f4bef821768d485ce17bf43: Status 404 returned error can't find the container with id 0fe3a687ff5cd38df1043ae82936deec15eb39486f4bef821768d485ce17bf43
Sep 08 16:51:10.144021 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:10.143986    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e43c7719-786tx" event={"ID":"06ede2fd-bb4a-431a-a29b-f574cc6f421a","Type":"ContainerStarted","Data":"0fe3a687ff5cd38df1043ae82936deec15eb39486f4bef821768d485ce17bf43"}
Sep 08 16:51:11.151375 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:11.151292    2557 generic.go:356] "Generic (PLEG): container finished" podID="06ede2fd-bb4a-431a-a29b-f574cc6f421a" containerID="b5db37f83c772e7d07b89b1664c9bae03aaff1c708d4de762848f97c6d4271da" exitCode=0
Sep 08 16:51:11.151729 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:11.151376    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e43c7719-786tx" event={"ID":"06ede2fd-bb4a-431a-a29b-f574cc6f421a","Type":"ContainerDied","Data":"b5db37f83c772e7d07b89b1664c9bae03aaff1c708d4de762848f97c6d4271da"}
Sep 08 16:51:12.292374 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:12.292352    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e43c7719-786tx"
Sep 08 16:51:12.413246 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:12.413223    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ede2fd-bb4a-431a-a29b-f574cc6f421a-maas-api-service-ca\") pod \"06ede2fd-bb4a-431a-a29b-f574cc6f421a\" (UID: \"06ede2fd-bb4a-431a-a29b-f574cc6f421a\") "
Sep 08 16:51:12.413576 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:12.413552    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ede2fd-bb4a-431a-a29b-f574cc6f421a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "06ede2fd-bb4a-431a-a29b-f574cc6f421a" (UID: "06ede2fd-bb4a-431a-a29b-f574cc6f421a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:51:12.514125 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:12.514099    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ede2fd-bb4a-431a-a29b-f574cc6f421a-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:51:13.164089 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:13.164063    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e43c7719-786tx"
Sep 08 16:51:13.164089 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:13.164072    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e43c7719-786tx" event={"ID":"06ede2fd-bb4a-431a-a29b-f574cc6f421a","Type":"ContainerDied","Data":"0fe3a687ff5cd38df1043ae82936deec15eb39486f4bef821768d485ce17bf43"}
Sep 08 16:51:13.164292 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:13.164112    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fe3a687ff5cd38df1043ae82936deec15eb39486f4bef821768d485ce17bf43"
Sep 08 16:51:15.434454 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:15.434418    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:51:16.476447 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:16.476418    2557 scope.go:117] "RemoveContainer" containerID="4fff6c1cea2bb2c759129e9987721ad38f975bdea7824b6f206aedf24439a164"
Sep 08 16:51:16.485131 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:16.485111    2557 scope.go:117] "RemoveContainer" containerID="00ddc0352ff58783054843f57645ba4db1bb577ec726bd2600cf41bae7462796"
Sep 08 16:51:16.492965 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:16.492944    2557 scope.go:117] "RemoveContainer" containerID="70c4ac6dedec8841ab738c76445434f26ca457068da7088f52ed9b5182ab7229"
Sep 08 16:51:25.202334 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:25.202302    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf"]
Sep 08 16:51:25.202808 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:25.202534    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf" podUID="38a2b5c4-76b0-42ac-9803-18fcc1989abe" containerName="payload-processing" containerID="cri-o://b5e18714679757201c1b36cd034b3cc70961571e6031547757f316b44f39adf2" gracePeriod=30
Sep 08 16:51:25.211017 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:25.210989    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr"]
Sep 08 16:51:25.211223 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:25.211201    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr" podUID="b03022d4-b768-4dc9-b20d-4bed68b0012e" containerName="payload-pre-processing" containerID="cri-o://782418eb1702c0b623ee427c9a36da471274e266ba365721f614b7fb418d4fa6" gracePeriod=30
Sep 08 16:51:25.432273 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:25.432241    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e43c7719-786tx"]
Sep 08 16:51:25.436780 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:25.436753    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e43c7719-786tx"]
Sep 08 16:51:25.503181 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:25.503158    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf"
Sep 08 16:51:25.506334 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:25.506314    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr"
Sep 08 16:51:25.631574 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:25.631547    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rl79\" (UniqueName: \"kubernetes.io/projected/38a2b5c4-76b0-42ac-9803-18fcc1989abe-kube-api-access-6rl79\") pod \"38a2b5c4-76b0-42ac-9803-18fcc1989abe\" (UID: \"38a2b5c4-76b0-42ac-9803-18fcc1989abe\") "
Sep 08 16:51:25.631764 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:25.631603    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqjd8\" (UniqueName: \"kubernetes.io/projected/b03022d4-b768-4dc9-b20d-4bed68b0012e-kube-api-access-gqjd8\") pod \"b03022d4-b768-4dc9-b20d-4bed68b0012e\" (UID: \"b03022d4-b768-4dc9-b20d-4bed68b0012e\") "
Sep 08 16:51:25.631764 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:25.631628    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a2b5c4-76b0-42ac-9803-18fcc1989abe-plugins-config-volume\") pod \"38a2b5c4-76b0-42ac-9803-18fcc1989abe\" (UID: \"38a2b5c4-76b0-42ac-9803-18fcc1989abe\") "
Sep 08 16:51:25.631764 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:25.631682    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b03022d4-b768-4dc9-b20d-4bed68b0012e-plugins-config-volume\") pod \"b03022d4-b768-4dc9-b20d-4bed68b0012e\" (UID: \"b03022d4-b768-4dc9-b20d-4bed68b0012e\") "
Sep 08 16:51:25.632048 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:25.632022    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a2b5c4-76b0-42ac-9803-18fcc1989abe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "38a2b5c4-76b0-42ac-9803-18fcc1989abe" (UID: "38a2b5c4-76b0-42ac-9803-18fcc1989abe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:51:25.632097 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:25.632055    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b03022d4-b768-4dc9-b20d-4bed68b0012e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b03022d4-b768-4dc9-b20d-4bed68b0012e" (UID: "b03022d4-b768-4dc9-b20d-4bed68b0012e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:51:25.633727 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:25.633707    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03022d4-b768-4dc9-b20d-4bed68b0012e-kube-api-access-gqjd8" (OuterVolumeSpecName: "kube-api-access-gqjd8") pod "b03022d4-b768-4dc9-b20d-4bed68b0012e" (UID: "b03022d4-b768-4dc9-b20d-4bed68b0012e"). InnerVolumeSpecName "kube-api-access-gqjd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:51:25.633778 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:25.633727    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a2b5c4-76b0-42ac-9803-18fcc1989abe-kube-api-access-6rl79" (OuterVolumeSpecName: "kube-api-access-6rl79") pod "38a2b5c4-76b0-42ac-9803-18fcc1989abe" (UID: "38a2b5c4-76b0-42ac-9803-18fcc1989abe"). InnerVolumeSpecName "kube-api-access-6rl79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:51:25.732620 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:25.732562    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rl79\" (UniqueName: \"kubernetes.io/projected/38a2b5c4-76b0-42ac-9803-18fcc1989abe-kube-api-access-6rl79\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:51:25.732620 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:25.732584    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqjd8\" (UniqueName: \"kubernetes.io/projected/b03022d4-b768-4dc9-b20d-4bed68b0012e-kube-api-access-gqjd8\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:51:25.732620 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:25.732594    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a2b5c4-76b0-42ac-9803-18fcc1989abe-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:51:25.732620 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:25.732603    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b03022d4-b768-4dc9-b20d-4bed68b0012e-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:51:26.247237 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:26.247204    2557 generic.go:356] "Generic (PLEG): container finished" podID="38a2b5c4-76b0-42ac-9803-18fcc1989abe" containerID="b5e18714679757201c1b36cd034b3cc70961571e6031547757f316b44f39adf2" exitCode=0
Sep 08 16:51:26.247586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:26.247250    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf"
Sep 08 16:51:26.247586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:26.247292    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf" event={"ID":"38a2b5c4-76b0-42ac-9803-18fcc1989abe","Type":"ContainerDied","Data":"b5e18714679757201c1b36cd034b3cc70961571e6031547757f316b44f39adf2"}
Sep 08 16:51:26.247586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:26.247347    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf" event={"ID":"38a2b5c4-76b0-42ac-9803-18fcc1989abe","Type":"ContainerDied","Data":"b9e55e43811abf308db95a36b524fd6fc630f78ce9ba896c8c750235313feb74"}
Sep 08 16:51:26.247586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:26.247369    2557 scope.go:117] "RemoveContainer" containerID="b5e18714679757201c1b36cd034b3cc70961571e6031547757f316b44f39adf2"
Sep 08 16:51:26.248754 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:26.248665    2557 generic.go:356] "Generic (PLEG): container finished" podID="b03022d4-b768-4dc9-b20d-4bed68b0012e" containerID="782418eb1702c0b623ee427c9a36da471274e266ba365721f614b7fb418d4fa6" exitCode=0
Sep 08 16:51:26.248754 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:26.248705    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr" event={"ID":"b03022d4-b768-4dc9-b20d-4bed68b0012e","Type":"ContainerDied","Data":"782418eb1702c0b623ee427c9a36da471274e266ba365721f614b7fb418d4fa6"}
Sep 08 16:51:26.248754 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:26.248745    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr" event={"ID":"b03022d4-b768-4dc9-b20d-4bed68b0012e","Type":"ContainerDied","Data":"39356a9a8502176baf9e82a85db4beecaaad5ff59f0b3a6e3e6bfea0bae930da"}
Sep 08 16:51:26.248960 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:26.248707    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr"
Sep 08 16:51:26.264236 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:26.264220    2557 scope.go:117] "RemoveContainer" containerID="b5e18714679757201c1b36cd034b3cc70961571e6031547757f316b44f39adf2"
Sep 08 16:51:26.264507 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:51:26.264490    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5e18714679757201c1b36cd034b3cc70961571e6031547757f316b44f39adf2\": container with ID starting with b5e18714679757201c1b36cd034b3cc70961571e6031547757f316b44f39adf2 not found: ID does not exist" containerID="b5e18714679757201c1b36cd034b3cc70961571e6031547757f316b44f39adf2"
Sep 08 16:51:26.264544 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:26.264513    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e18714679757201c1b36cd034b3cc70961571e6031547757f316b44f39adf2"} err="failed to get container status \"b5e18714679757201c1b36cd034b3cc70961571e6031547757f316b44f39adf2\": rpc error: code = NotFound desc = could not find container \"b5e18714679757201c1b36cd034b3cc70961571e6031547757f316b44f39adf2\": container with ID starting with b5e18714679757201c1b36cd034b3cc70961571e6031547757f316b44f39adf2 not found: ID does not exist"
Sep 08 16:51:26.264544 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:26.264527    2557 scope.go:117] "RemoveContainer" containerID="782418eb1702c0b623ee427c9a36da471274e266ba365721f614b7fb418d4fa6"
Sep 08 16:51:26.272998 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:26.272976    2557 scope.go:117] "RemoveContainer" containerID="782418eb1702c0b623ee427c9a36da471274e266ba365721f614b7fb418d4fa6"
Sep 08 16:51:26.273273 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:51:26.273256    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"782418eb1702c0b623ee427c9a36da471274e266ba365721f614b7fb418d4fa6\": container with ID starting with 782418eb1702c0b623ee427c9a36da471274e266ba365721f614b7fb418d4fa6 not found: ID does not exist" containerID="782418eb1702c0b623ee427c9a36da471274e266ba365721f614b7fb418d4fa6"
Sep 08 16:51:26.273338 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:26.273277    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"782418eb1702c0b623ee427c9a36da471274e266ba365721f614b7fb418d4fa6"} err="failed to get container status \"782418eb1702c0b623ee427c9a36da471274e266ba365721f614b7fb418d4fa6\": rpc error: code = NotFound desc = could not find container \"782418eb1702c0b623ee427c9a36da471274e266ba365721f614b7fb418d4fa6\": container with ID starting with 782418eb1702c0b623ee427c9a36da471274e266ba365721f614b7fb418d4fa6 not found: ID does not exist"
Sep 08 16:51:26.277321 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:26.277299    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf"]
Sep 08 16:51:26.279582 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:26.279565    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e43c7719-5558fd8f4f-cpzhf"]
Sep 08 16:51:26.289876 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:26.289857    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr"]
Sep 08 16:51:26.293732 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:26.293714    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e43c7719-5b4fb56494-dmtsr"]
Sep 08 16:51:26.346688 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:26.346656    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06ede2fd-bb4a-431a-a29b-f574cc6f421a" path="/var/lib/kubelet/pods/06ede2fd-bb4a-431a-a29b-f574cc6f421a/volumes"
Sep 08 16:51:26.347053 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:26.347038    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38a2b5c4-76b0-42ac-9803-18fcc1989abe" path="/var/lib/kubelet/pods/38a2b5c4-76b0-42ac-9803-18fcc1989abe/volumes"
Sep 08 16:51:26.347367 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:26.347355    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b03022d4-b768-4dc9-b20d-4bed68b0012e" path="/var/lib/kubelet/pods/b03022d4-b768-4dc9-b20d-4bed68b0012e/volumes"
Sep 08 16:51:30.333910 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:30.333874    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:51:33.670642 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:33.670604    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"]
Sep 08 16:51:46.238028 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:46.237991    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:51:48.690258 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:48.690187    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w" podUID="dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd" containerName="main" containerID="cri-o://ccaa5be2e1dfc0bd5da11bba06b8829609907d2202700a5bc472499032058788" gracePeriod=45
Sep 08 16:51:48.944616 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:48.944564    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:51:49.031427 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.031396    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-tls-certs\") pod \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\" (UID: \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\") "
Sep 08 16:51:49.031427 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.031429    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hxgf\" (UniqueName: \"kubernetes.io/projected/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-kube-api-access-6hxgf\") pod \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\" (UID: \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\") "
Sep 08 16:51:49.031708 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.031451    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-model-cache\") pod \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\" (UID: \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\") "
Sep 08 16:51:49.031708 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.031468    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-dshm\") pod \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\" (UID: \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\") "
Sep 08 16:51:49.031708 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.031521    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-tmp-dir\") pod \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\" (UID: \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\") "
Sep 08 16:51:49.031708 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.031576    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-home\") pod \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\" (UID: \"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd\") "
Sep 08 16:51:49.031962 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.031760    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd" (UID: "dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:51:49.031962 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.031815    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd" (UID: "dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:51:49.031962 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.031904    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:51:49.031962 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.031922    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:51:49.032193 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.031961    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-home" (OuterVolumeSpecName: "home") pod "dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd" (UID: "dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:51:49.033949 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.033919    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-dshm" (OuterVolumeSpecName: "dshm") pod "dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd" (UID: "dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:51:49.034072 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.034004    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd" (UID: "dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:51:49.034203 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.034178    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-kube-api-access-6hxgf" (OuterVolumeSpecName: "kube-api-access-6hxgf") pod "dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd" (UID: "dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd"). InnerVolumeSpecName "kube-api-access-6hxgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:51:49.133250 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.133224    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:51:49.133250 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.133248    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:51:49.133411 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.133258    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hxgf\" (UniqueName: \"kubernetes.io/projected/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-kube-api-access-6hxgf\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:51:49.133411 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.133266    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:51:49.369550 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.369478    2557 generic.go:356] "Generic (PLEG): container finished" podID="dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd" containerID="ccaa5be2e1dfc0bd5da11bba06b8829609907d2202700a5bc472499032058788" exitCode=0
Sep 08 16:51:49.369550 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.369533    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"
Sep 08 16:51:49.369715 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.369560    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w" event={"ID":"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd","Type":"ContainerDied","Data":"ccaa5be2e1dfc0bd5da11bba06b8829609907d2202700a5bc472499032058788"}
Sep 08 16:51:49.369715 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.369596    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w" event={"ID":"dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd","Type":"ContainerDied","Data":"49017e7dc5d1a8a4e32f554f3262b421342f874cb955694664ee8a52671745eb"}
Sep 08 16:51:49.369715 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.369612    2557 scope.go:117] "RemoveContainer" containerID="ccaa5be2e1dfc0bd5da11bba06b8829609907d2202700a5bc472499032058788"
Sep 08 16:51:49.378755 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.378734    2557 scope.go:117] "RemoveContainer" containerID="ccaa5be2e1dfc0bd5da11bba06b8829609907d2202700a5bc472499032058788"
Sep 08 16:51:49.379050 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:51:49.379033    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccaa5be2e1dfc0bd5da11bba06b8829609907d2202700a5bc472499032058788\": container with ID starting with ccaa5be2e1dfc0bd5da11bba06b8829609907d2202700a5bc472499032058788 not found: ID does not exist" containerID="ccaa5be2e1dfc0bd5da11bba06b8829609907d2202700a5bc472499032058788"
Sep 08 16:51:49.379102 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.379058    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccaa5be2e1dfc0bd5da11bba06b8829609907d2202700a5bc472499032058788"} err="failed to get container status \"ccaa5be2e1dfc0bd5da11bba06b8829609907d2202700a5bc472499032058788\": rpc error: code = NotFound desc = could not find container \"ccaa5be2e1dfc0bd5da11bba06b8829609907d2202700a5bc472499032058788\": container with ID starting with ccaa5be2e1dfc0bd5da11bba06b8829609907d2202700a5bc472499032058788 not found: ID does not exist"
Sep 08 16:51:49.390076 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.390051    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"]
Sep 08 16:51:49.394561 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:49.394544    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-e43c7719/e2e-lifecycle-model-e43c7719-kserve-774bb56dbc-d7l6w"]
Sep 08 16:51:50.345613 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:51:50.345580    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd" path="/var/lib/kubelet/pods/dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd/volumes"
Sep 08 16:52:01.426236 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:01.426201    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:52:15.577405 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:15.577368    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"]
Sep 08 16:52:15.803843 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:15.803802    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:52:16.472583 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:16.472556    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"]
Sep 08 16:52:16.519196 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:16.519173    2557 scope.go:117] "RemoveContainer" containerID="8414a7c1f79a0bf3360933e861173972e565e64a8dd5111c78d136f087441fe1"
Sep 08 16:52:16.526961 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:16.526945    2557 scope.go:117] "RemoveContainer" containerID="83732e6fd1a9cfffaa3c527e8c6e58f19f1d49c6bd171d8fcf4f2796f36d27c1"
Sep 08 16:52:16.534436 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:16.534414    2557 scope.go:117] "RemoveContainer" containerID="e10c79ce0382bdfce579c4c18ac5661870c670ec8497b031faf8a485fe6b70c4"
Sep 08 16:52:23.260026 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.259995    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"]
Sep 08 16:52:23.260451 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.260378    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b03022d4-b768-4dc9-b20d-4bed68b0012e" containerName="payload-pre-processing"
Sep 08 16:52:23.260451 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.260392    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b03022d4-b768-4dc9-b20d-4bed68b0012e" containerName="payload-pre-processing"
Sep 08 16:52:23.260451 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.260410    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38a2b5c4-76b0-42ac-9803-18fcc1989abe" containerName="payload-processing"
Sep 08 16:52:23.260451 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.260415    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="38a2b5c4-76b0-42ac-9803-18fcc1989abe" containerName="payload-processing"
Sep 08 16:52:23.260451 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.260429    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06ede2fd-bb4a-431a-a29b-f574cc6f421a" containerName="revoke-keys"
Sep 08 16:52:23.260451 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.260434    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="06ede2fd-bb4a-431a-a29b-f574cc6f421a" containerName="revoke-keys"
Sep 08 16:52:23.260451 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.260443    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd" containerName="main"
Sep 08 16:52:23.260451 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.260447    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd" containerName="main"
Sep 08 16:52:23.260690 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.260500    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b03022d4-b768-4dc9-b20d-4bed68b0012e" containerName="payload-pre-processing"
Sep 08 16:52:23.260690 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.260506    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="38a2b5c4-76b0-42ac-9803-18fcc1989abe" containerName="payload-processing"
Sep 08 16:52:23.260690 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.260515    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbdaa4b6-303a-4dd5-a0f7-be074d0c51bd" containerName="main"
Sep 08 16:52:23.260690 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.260519    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="06ede2fd-bb4a-431a-a29b-f574cc6f421a" containerName="revoke-keys"
Sep 08 16:52:23.264820 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.264804    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:52:23.268205 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.268184    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-608268\"/\"sub-test-model-608268-kserve-self-signed-certs\""
Sep 08 16:52:23.290010 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.289979    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"]
Sep 08 16:52:23.416134 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.416107    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a89f53-df8c-4464-8a1b-fa6907e3db86-tmp-dir\") pod \"sub-test-model-608268-kserve-5c74cff87f-rtc6z\" (UID: \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\") " pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:52:23.416269 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.416153    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a89f53-df8c-4464-8a1b-fa6907e3db86-dshm\") pod \"sub-test-model-608268-kserve-5c74cff87f-rtc6z\" (UID: \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\") " pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:52:23.416269 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.416203    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3a89f53-df8c-4464-8a1b-fa6907e3db86-home\") pod \"sub-test-model-608268-kserve-5c74cff87f-rtc6z\" (UID: \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\") " pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:52:23.416269 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.416240    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a89f53-df8c-4464-8a1b-fa6907e3db86-tls-certs\") pod \"sub-test-model-608268-kserve-5c74cff87f-rtc6z\" (UID: \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\") " pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:52:23.416370 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.416296    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s8xx\" (UniqueName: \"kubernetes.io/projected/d3a89f53-df8c-4464-8a1b-fa6907e3db86-kube-api-access-2s8xx\") pod \"sub-test-model-608268-kserve-5c74cff87f-rtc6z\" (UID: \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\") " pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:52:23.416370 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.416350    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a89f53-df8c-4464-8a1b-fa6907e3db86-model-cache\") pod \"sub-test-model-608268-kserve-5c74cff87f-rtc6z\" (UID: \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\") " pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:52:23.517353 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.517281    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a89f53-df8c-4464-8a1b-fa6907e3db86-model-cache\") pod \"sub-test-model-608268-kserve-5c74cff87f-rtc6z\" (UID: \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\") " pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:52:23.517353 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.517326    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a89f53-df8c-4464-8a1b-fa6907e3db86-tmp-dir\") pod \"sub-test-model-608268-kserve-5c74cff87f-rtc6z\" (UID: \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\") " pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:52:23.517543 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.517382    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a89f53-df8c-4464-8a1b-fa6907e3db86-dshm\") pod \"sub-test-model-608268-kserve-5c74cff87f-rtc6z\" (UID: \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\") " pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:52:23.517543 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.517408    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3a89f53-df8c-4464-8a1b-fa6907e3db86-home\") pod \"sub-test-model-608268-kserve-5c74cff87f-rtc6z\" (UID: \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\") " pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:52:23.517543 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.517441    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a89f53-df8c-4464-8a1b-fa6907e3db86-tls-certs\") pod \"sub-test-model-608268-kserve-5c74cff87f-rtc6z\" (UID: \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\") " pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:52:23.517543 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.517488    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s8xx\" (UniqueName: \"kubernetes.io/projected/d3a89f53-df8c-4464-8a1b-fa6907e3db86-kube-api-access-2s8xx\") pod \"sub-test-model-608268-kserve-5c74cff87f-rtc6z\" (UID: \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\") " pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:52:23.517747 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.517698    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a89f53-df8c-4464-8a1b-fa6907e3db86-model-cache\") pod \"sub-test-model-608268-kserve-5c74cff87f-rtc6z\" (UID: \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\") " pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:52:23.517807 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.517775    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a89f53-df8c-4464-8a1b-fa6907e3db86-tmp-dir\") pod \"sub-test-model-608268-kserve-5c74cff87f-rtc6z\" (UID: \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\") " pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:52:23.517911 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.517888    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3a89f53-df8c-4464-8a1b-fa6907e3db86-home\") pod \"sub-test-model-608268-kserve-5c74cff87f-rtc6z\" (UID: \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\") " pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:52:23.519556 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.519533    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a89f53-df8c-4464-8a1b-fa6907e3db86-dshm\") pod \"sub-test-model-608268-kserve-5c74cff87f-rtc6z\" (UID: \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\") " pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:52:23.519938 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.519916    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a89f53-df8c-4464-8a1b-fa6907e3db86-tls-certs\") pod \"sub-test-model-608268-kserve-5c74cff87f-rtc6z\" (UID: \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\") " pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:52:23.526629 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.526608    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s8xx\" (UniqueName: \"kubernetes.io/projected/d3a89f53-df8c-4464-8a1b-fa6907e3db86-kube-api-access-2s8xx\") pod \"sub-test-model-608268-kserve-5c74cff87f-rtc6z\" (UID: \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\") " pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:52:23.574254 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.574227    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:52:23.861527 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:23.861501    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"]
Sep 08 16:52:23.863257 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:52:23.863232    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3a89f53_df8c_4464_8a1b_fa6907e3db86.slice/crio-ba168f49cf9c91af816dba513f159cc7ff93a352d56acfd91dde2180387f90d1 WatchSource:0}: Error finding container ba168f49cf9c91af816dba513f159cc7ff93a352d56acfd91dde2180387f90d1: Status 404 returned error can't find the container with id ba168f49cf9c91af816dba513f159cc7ff93a352d56acfd91dde2180387f90d1
Sep 08 16:52:24.553283 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:24.553246    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z" event={"ID":"d3a89f53-df8c-4464-8a1b-fa6907e3db86","Type":"ContainerStarted","Data":"579bcda92adf66177e5c96763082a0c8e4b78b94e0ba9134234b7a8283509298"}
Sep 08 16:52:24.553628 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:24.553294    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z" event={"ID":"d3a89f53-df8c-4464-8a1b-fa6907e3db86","Type":"ContainerStarted","Data":"ba168f49cf9c91af816dba513f159cc7ff93a352d56acfd91dde2180387f90d1"}
Sep 08 16:52:24.577613 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:24.577565    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z" podStartSLOduration=1.577551357 podStartE2EDuration="1.577551357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:24.575874303 +0000 UTC m=+1328.738845750" watchObservedRunningTime="2026-09-08 16:52:24.577551357 +0000 UTC m=+1328.740522803"
Sep 08 16:52:29.873892 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:29.870242    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:52:30.596494 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:30.596454    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk" podUID="37059f8f-c983-4786-8836-2f07d17dc8cf" containerName="main" containerID="cri-o://6f5d22ec9721e02584a9629b22040d5cb941aceeb2d84d0c5074e554485a19cb" gracePeriod=45
Sep 08 16:52:30.846234 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:30.846209    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:52:30.980040 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:30.980011    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37059f8f-c983-4786-8836-2f07d17dc8cf-tls-certs\") pod \"37059f8f-c983-4786-8836-2f07d17dc8cf\" (UID: \"37059f8f-c983-4786-8836-2f07d17dc8cf\") "
Sep 08 16:52:30.980440 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:30.980043    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37059f8f-c983-4786-8836-2f07d17dc8cf-home\") pod \"37059f8f-c983-4786-8836-2f07d17dc8cf\" (UID: \"37059f8f-c983-4786-8836-2f07d17dc8cf\") "
Sep 08 16:52:30.980440 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:30.980084    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55zm9\" (UniqueName: \"kubernetes.io/projected/37059f8f-c983-4786-8836-2f07d17dc8cf-kube-api-access-55zm9\") pod \"37059f8f-c983-4786-8836-2f07d17dc8cf\" (UID: \"37059f8f-c983-4786-8836-2f07d17dc8cf\") "
Sep 08 16:52:30.980440 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:30.980165    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37059f8f-c983-4786-8836-2f07d17dc8cf-model-cache\") pod \"37059f8f-c983-4786-8836-2f07d17dc8cf\" (UID: \"37059f8f-c983-4786-8836-2f07d17dc8cf\") "
Sep 08 16:52:30.980440 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:30.980212    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37059f8f-c983-4786-8836-2f07d17dc8cf-tmp-dir\") pod \"37059f8f-c983-4786-8836-2f07d17dc8cf\" (UID: \"37059f8f-c983-4786-8836-2f07d17dc8cf\") "
Sep 08 16:52:30.980440 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:30.980239    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37059f8f-c983-4786-8836-2f07d17dc8cf-dshm\") pod \"37059f8f-c983-4786-8836-2f07d17dc8cf\" (UID: \"37059f8f-c983-4786-8836-2f07d17dc8cf\") "
Sep 08 16:52:30.980440 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:30.980307    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37059f8f-c983-4786-8836-2f07d17dc8cf-home" (OuterVolumeSpecName: "home") pod "37059f8f-c983-4786-8836-2f07d17dc8cf" (UID: "37059f8f-c983-4786-8836-2f07d17dc8cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:52:30.980695 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:30.980441    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37059f8f-c983-4786-8836-2f07d17dc8cf-model-cache" (OuterVolumeSpecName: "model-cache") pod "37059f8f-c983-4786-8836-2f07d17dc8cf" (UID: "37059f8f-c983-4786-8836-2f07d17dc8cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:52:30.980695 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:30.980461    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37059f8f-c983-4786-8836-2f07d17dc8cf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37059f8f-c983-4786-8836-2f07d17dc8cf" (UID: "37059f8f-c983-4786-8836-2f07d17dc8cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:52:30.980695 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:30.980544    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37059f8f-c983-4786-8836-2f07d17dc8cf-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:52:30.980695 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:30.980561    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37059f8f-c983-4786-8836-2f07d17dc8cf-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:52:30.980695 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:30.980575    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37059f8f-c983-4786-8836-2f07d17dc8cf-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:52:30.982400 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:30.982379    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37059f8f-c983-4786-8836-2f07d17dc8cf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37059f8f-c983-4786-8836-2f07d17dc8cf" (UID: "37059f8f-c983-4786-8836-2f07d17dc8cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:52:30.982400 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:30.982380    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37059f8f-c983-4786-8836-2f07d17dc8cf-kube-api-access-55zm9" (OuterVolumeSpecName: "kube-api-access-55zm9") pod "37059f8f-c983-4786-8836-2f07d17dc8cf" (UID: "37059f8f-c983-4786-8836-2f07d17dc8cf"). InnerVolumeSpecName "kube-api-access-55zm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:52:30.982400 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:30.982380    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37059f8f-c983-4786-8836-2f07d17dc8cf-dshm" (OuterVolumeSpecName: "dshm") pod "37059f8f-c983-4786-8836-2f07d17dc8cf" (UID: "37059f8f-c983-4786-8836-2f07d17dc8cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:52:31.081156 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.081133    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55zm9\" (UniqueName: \"kubernetes.io/projected/37059f8f-c983-4786-8836-2f07d17dc8cf-kube-api-access-55zm9\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:52:31.081156 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.081154    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37059f8f-c983-4786-8836-2f07d17dc8cf-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:52:31.081314 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.081165    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37059f8f-c983-4786-8836-2f07d17dc8cf-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:52:31.492179 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.492117    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74" podUID="54965c1c-2b57-4ec0-bbee-9115db00bb0b" containerName="main" containerID="cri-o://435a59f93b6bdd9bbb89ccf0ecb83c256f9e95b95ed648dfb2f0fe2a71906588" gracePeriod=45
Sep 08 16:52:31.591460 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.591434    2557 generic.go:356] "Generic (PLEG): container finished" podID="37059f8f-c983-4786-8836-2f07d17dc8cf" containerID="6f5d22ec9721e02584a9629b22040d5cb941aceeb2d84d0c5074e554485a19cb" exitCode=0
Sep 08 16:52:31.591594 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.591473    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"
Sep 08 16:52:31.591594 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.591530    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk" event={"ID":"37059f8f-c983-4786-8836-2f07d17dc8cf","Type":"ContainerDied","Data":"6f5d22ec9721e02584a9629b22040d5cb941aceeb2d84d0c5074e554485a19cb"}
Sep 08 16:52:31.591594 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.591573    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk" event={"ID":"37059f8f-c983-4786-8836-2f07d17dc8cf","Type":"ContainerDied","Data":"22a5a52244621cdaba337bbbe06ace7171801901513d904850ff7403d508aa17"}
Sep 08 16:52:31.591594 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.591589    2557 scope.go:117] "RemoveContainer" containerID="6f5d22ec9721e02584a9629b22040d5cb941aceeb2d84d0c5074e554485a19cb"
Sep 08 16:52:31.603476 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.603333    2557 scope.go:117] "RemoveContainer" containerID="6f5d22ec9721e02584a9629b22040d5cb941aceeb2d84d0c5074e554485a19cb"
Sep 08 16:52:31.603627 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:52:31.603610    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f5d22ec9721e02584a9629b22040d5cb941aceeb2d84d0c5074e554485a19cb\": container with ID starting with 6f5d22ec9721e02584a9629b22040d5cb941aceeb2d84d0c5074e554485a19cb not found: ID does not exist" containerID="6f5d22ec9721e02584a9629b22040d5cb941aceeb2d84d0c5074e554485a19cb"
Sep 08 16:52:31.603671 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.603635    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f5d22ec9721e02584a9629b22040d5cb941aceeb2d84d0c5074e554485a19cb"} err="failed to get container status \"6f5d22ec9721e02584a9629b22040d5cb941aceeb2d84d0c5074e554485a19cb\": rpc error: code = NotFound desc = could not find container \"6f5d22ec9721e02584a9629b22040d5cb941aceeb2d84d0c5074e554485a19cb\": container with ID starting with 6f5d22ec9721e02584a9629b22040d5cb941aceeb2d84d0c5074e554485a19cb not found: ID does not exist"
Sep 08 16:52:31.618890 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.618868    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"]
Sep 08 16:52:31.628465 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.628432    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-608268/auth-test-model-608268-kserve-7586d5f75f-8d6vk"]
Sep 08 16:52:31.742820 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.742772    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:52:31.887865 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.887819    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54965c1c-2b57-4ec0-bbee-9115db00bb0b-model-cache\") pod \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\" (UID: \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\") "
Sep 08 16:52:31.888025 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.887893    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54965c1c-2b57-4ec0-bbee-9115db00bb0b-tmp-dir\") pod \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\" (UID: \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\") "
Sep 08 16:52:31.888025 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.887938    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54965c1c-2b57-4ec0-bbee-9115db00bb0b-home\") pod \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\" (UID: \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\") "
Sep 08 16:52:31.888025 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.887959    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54965c1c-2b57-4ec0-bbee-9115db00bb0b-dshm\") pod \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\" (UID: \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\") "
Sep 08 16:52:31.888025 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.888007    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wk66r\" (UniqueName: \"kubernetes.io/projected/54965c1c-2b57-4ec0-bbee-9115db00bb0b-kube-api-access-wk66r\") pod \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\" (UID: \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\") "
Sep 08 16:52:31.888227 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.888040    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54965c1c-2b57-4ec0-bbee-9115db00bb0b-tls-certs\") pod \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\" (UID: \"54965c1c-2b57-4ec0-bbee-9115db00bb0b\") "
Sep 08 16:52:31.888227 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.888131    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54965c1c-2b57-4ec0-bbee-9115db00bb0b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "54965c1c-2b57-4ec0-bbee-9115db00bb0b" (UID: "54965c1c-2b57-4ec0-bbee-9115db00bb0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:52:31.888227 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.888144    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54965c1c-2b57-4ec0-bbee-9115db00bb0b-model-cache" (OuterVolumeSpecName: "model-cache") pod "54965c1c-2b57-4ec0-bbee-9115db00bb0b" (UID: "54965c1c-2b57-4ec0-bbee-9115db00bb0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:52:31.888227 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.888168    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54965c1c-2b57-4ec0-bbee-9115db00bb0b-home" (OuterVolumeSpecName: "home") pod "54965c1c-2b57-4ec0-bbee-9115db00bb0b" (UID: "54965c1c-2b57-4ec0-bbee-9115db00bb0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:52:31.888416 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.888316    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54965c1c-2b57-4ec0-bbee-9115db00bb0b-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:52:31.888416 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.888328    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54965c1c-2b57-4ec0-bbee-9115db00bb0b-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:52:31.888416 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.888346    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54965c1c-2b57-4ec0-bbee-9115db00bb0b-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:52:31.890291 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.890271    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54965c1c-2b57-4ec0-bbee-9115db00bb0b-dshm" (OuterVolumeSpecName: "dshm") pod "54965c1c-2b57-4ec0-bbee-9115db00bb0b" (UID: "54965c1c-2b57-4ec0-bbee-9115db00bb0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:52:31.890291 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.890276    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54965c1c-2b57-4ec0-bbee-9115db00bb0b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54965c1c-2b57-4ec0-bbee-9115db00bb0b" (UID: "54965c1c-2b57-4ec0-bbee-9115db00bb0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:52:31.890437 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.890383    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54965c1c-2b57-4ec0-bbee-9115db00bb0b-kube-api-access-wk66r" (OuterVolumeSpecName: "kube-api-access-wk66r") pod "54965c1c-2b57-4ec0-bbee-9115db00bb0b" (UID: "54965c1c-2b57-4ec0-bbee-9115db00bb0b"). InnerVolumeSpecName "kube-api-access-wk66r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:52:31.989205 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.989177    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54965c1c-2b57-4ec0-bbee-9115db00bb0b-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:52:31.989205 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.989207    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk66r\" (UniqueName: \"kubernetes.io/projected/54965c1c-2b57-4ec0-bbee-9115db00bb0b-kube-api-access-wk66r\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:52:31.989526 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:31.989220    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54965c1c-2b57-4ec0-bbee-9115db00bb0b-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:52:32.346200 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:32.346155    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37059f8f-c983-4786-8836-2f07d17dc8cf" path="/var/lib/kubelet/pods/37059f8f-c983-4786-8836-2f07d17dc8cf/volumes"
Sep 08 16:52:32.598583 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:32.598500    2557 generic.go:356] "Generic (PLEG): container finished" podID="54965c1c-2b57-4ec0-bbee-9115db00bb0b" containerID="435a59f93b6bdd9bbb89ccf0ecb83c256f9e95b95ed648dfb2f0fe2a71906588" exitCode=0
Sep 08 16:52:32.598583 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:32.598544    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"
Sep 08 16:52:32.598771 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:32.598587    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74" event={"ID":"54965c1c-2b57-4ec0-bbee-9115db00bb0b","Type":"ContainerDied","Data":"435a59f93b6bdd9bbb89ccf0ecb83c256f9e95b95ed648dfb2f0fe2a71906588"}
Sep 08 16:52:32.598771 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:32.598626    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74" event={"ID":"54965c1c-2b57-4ec0-bbee-9115db00bb0b","Type":"ContainerDied","Data":"edbd4f57f2cb5b72bed04b2e3c6e2075384dd47af2e0ed3e7a69d2ea3897f985"}
Sep 08 16:52:32.598771 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:32.598647    2557 scope.go:117] "RemoveContainer" containerID="435a59f93b6bdd9bbb89ccf0ecb83c256f9e95b95ed648dfb2f0fe2a71906588"
Sep 08 16:52:32.607202 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:32.607035    2557 scope.go:117] "RemoveContainer" containerID="435a59f93b6bdd9bbb89ccf0ecb83c256f9e95b95ed648dfb2f0fe2a71906588"
Sep 08 16:52:32.607348 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:52:32.607326    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"435a59f93b6bdd9bbb89ccf0ecb83c256f9e95b95ed648dfb2f0fe2a71906588\": container with ID starting with 435a59f93b6bdd9bbb89ccf0ecb83c256f9e95b95ed648dfb2f0fe2a71906588 not found: ID does not exist" containerID="435a59f93b6bdd9bbb89ccf0ecb83c256f9e95b95ed648dfb2f0fe2a71906588"
Sep 08 16:52:32.607385 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:32.607359    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"435a59f93b6bdd9bbb89ccf0ecb83c256f9e95b95ed648dfb2f0fe2a71906588"} err="failed to get container status \"435a59f93b6bdd9bbb89ccf0ecb83c256f9e95b95ed648dfb2f0fe2a71906588\": rpc error: code = NotFound desc = could not find container \"435a59f93b6bdd9bbb89ccf0ecb83c256f9e95b95ed648dfb2f0fe2a71906588\": container with ID starting with 435a59f93b6bdd9bbb89ccf0ecb83c256f9e95b95ed648dfb2f0fe2a71906588 not found: ID does not exist"
Sep 08 16:52:32.621395 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:32.621374    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"]
Sep 08 16:52:32.624798 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:32.624778    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6b15ca/auth-test-model-6b15ca-kserve-587578f459-mws74"]
Sep 08 16:52:33.575409 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:33.575371    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:52:33.575754 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:33.575421    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:52:33.588703 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:33.588683    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:52:33.624077 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:33.624053    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:52:34.345474 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:34.345439    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54965c1c-2b57-4ec0-bbee-9115db00bb0b" path="/var/lib/kubelet/pods/54965c1c-2b57-4ec0-bbee-9115db00bb0b/volumes"
Sep 08 16:52:40.352444 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:40.352354    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:52:45.654653 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.654620    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"]
Sep 08 16:52:45.655108 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.655012    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54965c1c-2b57-4ec0-bbee-9115db00bb0b" containerName="main"
Sep 08 16:52:45.655108 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.655025    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="54965c1c-2b57-4ec0-bbee-9115db00bb0b" containerName="main"
Sep 08 16:52:45.655108 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.655050    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37059f8f-c983-4786-8836-2f07d17dc8cf" containerName="main"
Sep 08 16:52:45.655108 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.655060    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="37059f8f-c983-4786-8836-2f07d17dc8cf" containerName="main"
Sep 08 16:52:45.655322 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.655115    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="54965c1c-2b57-4ec0-bbee-9115db00bb0b" containerName="main"
Sep 08 16:52:45.655322 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.655125    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="37059f8f-c983-4786-8836-2f07d17dc8cf" containerName="main"
Sep 08 16:52:45.657584 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.657567    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:52:45.660027 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.660007    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6b15ca\"/\"openshift-service-ca.crt\""
Sep 08 16:52:45.660145 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.660121    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6b15ca\"/\"default-dockercfg-c8vzm\""
Sep 08 16:52:45.660996 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.660980    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6b15ca\"/\"kube-root-ca.crt\""
Sep 08 16:52:45.661058 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.661042    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6b15ca\"/\"sub-test-model-6b15ca-kserve-self-signed-certs\""
Sep 08 16:52:45.680351 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.680326    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"]
Sep 08 16:52:45.700977 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.700954    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-model-cache\") pod \"sub-test-model-6b15ca-kserve-578f88b859-txdzr\" (UID: \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:52:45.701095 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.700988    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-dshm\") pod \"sub-test-model-6b15ca-kserve-578f88b859-txdzr\" (UID: \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:52:45.701095 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.701004    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgwkh\" (UniqueName: \"kubernetes.io/projected/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-kube-api-access-zgwkh\") pod \"sub-test-model-6b15ca-kserve-578f88b859-txdzr\" (UID: \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:52:45.701095 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.701024    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-tls-certs\") pod \"sub-test-model-6b15ca-kserve-578f88b859-txdzr\" (UID: \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:52:45.701095 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.701082    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-tmp-dir\") pod \"sub-test-model-6b15ca-kserve-578f88b859-txdzr\" (UID: \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:52:45.701283 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.701142    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-home\") pod \"sub-test-model-6b15ca-kserve-578f88b859-txdzr\" (UID: \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:52:45.801634 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.801604    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-model-cache\") pod \"sub-test-model-6b15ca-kserve-578f88b859-txdzr\" (UID: \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:52:45.801775 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.801670    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-dshm\") pod \"sub-test-model-6b15ca-kserve-578f88b859-txdzr\" (UID: \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:52:45.801775 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.801694    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgwkh\" (UniqueName: \"kubernetes.io/projected/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-kube-api-access-zgwkh\") pod \"sub-test-model-6b15ca-kserve-578f88b859-txdzr\" (UID: \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:52:45.801775 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.801718    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-tls-certs\") pod \"sub-test-model-6b15ca-kserve-578f88b859-txdzr\" (UID: \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:52:45.801775 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.801743    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-tmp-dir\") pod \"sub-test-model-6b15ca-kserve-578f88b859-txdzr\" (UID: \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:52:45.802055 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.801855    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-home\") pod \"sub-test-model-6b15ca-kserve-578f88b859-txdzr\" (UID: \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:52:45.802114 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.802081    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-model-cache\") pod \"sub-test-model-6b15ca-kserve-578f88b859-txdzr\" (UID: \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:52:45.802264 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.802240    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-home\") pod \"sub-test-model-6b15ca-kserve-578f88b859-txdzr\" (UID: \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:52:45.802446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.802308    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-tmp-dir\") pod \"sub-test-model-6b15ca-kserve-578f88b859-txdzr\" (UID: \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:52:45.804258 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.804238    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-dshm\") pod \"sub-test-model-6b15ca-kserve-578f88b859-txdzr\" (UID: \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:52:45.804554 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.804537    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-tls-certs\") pod \"sub-test-model-6b15ca-kserve-578f88b859-txdzr\" (UID: \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:52:45.818073 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.818047    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgwkh\" (UniqueName: \"kubernetes.io/projected/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-kube-api-access-zgwkh\") pod \"sub-test-model-6b15ca-kserve-578f88b859-txdzr\" (UID: \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:52:45.966936 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:45.966904    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:52:46.245747 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:46.245721    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"]
Sep 08 16:52:46.247286 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:52:46.247260    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b0a3597_fb4e_4b54_b3d3_60c194c07caf.slice/crio-70c1211a246acfe231f97743dc7fb0868e8cb7abcc4dcaebf594ed9f1f35e4a1 WatchSource:0}: Error finding container 70c1211a246acfe231f97743dc7fb0868e8cb7abcc4dcaebf594ed9f1f35e4a1: Status 404 returned error can't find the container with id 70c1211a246acfe231f97743dc7fb0868e8cb7abcc4dcaebf594ed9f1f35e4a1
Sep 08 16:52:46.676530 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:46.676498    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr" event={"ID":"9b0a3597-fb4e-4b54-b3d3-60c194c07caf","Type":"ContainerStarted","Data":"a4c553df45f490285fdbe05a0464e516d715eddb110e32ae886da12932f686de"}
Sep 08 16:52:46.677012 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:46.676541    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr" event={"ID":"9b0a3597-fb4e-4b54-b3d3-60c194c07caf","Type":"ContainerStarted","Data":"70c1211a246acfe231f97743dc7fb0868e8cb7abcc4dcaebf594ed9f1f35e4a1"}
Sep 08 16:52:46.703555 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:46.703513    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr" podStartSLOduration=1.703500566 podStartE2EDuration="1.703500566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:46.702426336 +0000 UTC m=+1350.865397781" watchObservedRunningTime="2026-09-08 16:52:46.703500566 +0000 UTC m=+1350.866472013"
Sep 08 16:52:50.789734 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:50.789700    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:52:55.967905 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:55.967871    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:52:55.968362 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:55.967924    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:52:55.980084 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:55.980064    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:52:56.738038 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:52:56.738008    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:53:09.254207 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:09.254174    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:53:16.568516 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:16.568487    2557 scope.go:117] "RemoveContainer" containerID="c63ba6bcc0d3191c30c1455bd372adeb43dc115fd8e7c883e841a80482f421f3"
Sep 08 16:53:21.629878 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:21.629848    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk"]
Sep 08 16:53:21.634826 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:21.634807    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk"
Sep 08 16:53:21.637549 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:21.637531    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-246729-dockercfg-nmsjp\""
Sep 08 16:53:21.637796 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:21.637781    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-246729\""
Sep 08 16:53:21.662032 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:21.662012    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk"]
Sep 08 16:53:21.800528 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:21.800490    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/866bb25b-8cde-40b9-829d-40abdcf040b0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk\" (UID: \"866bb25b-8cde-40b9-829d-40abdcf040b0\") " pod="openshift-ingress/payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk"
Sep 08 16:53:21.800669 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:21.800550    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq7lk\" (UniqueName: \"kubernetes.io/projected/866bb25b-8cde-40b9-829d-40abdcf040b0-kube-api-access-tq7lk\") pod \"payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk\" (UID: \"866bb25b-8cde-40b9-829d-40abdcf040b0\") " pod="openshift-ingress/payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk"
Sep 08 16:53:21.901222 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:21.901153    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq7lk\" (UniqueName: \"kubernetes.io/projected/866bb25b-8cde-40b9-829d-40abdcf040b0-kube-api-access-tq7lk\") pod \"payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk\" (UID: \"866bb25b-8cde-40b9-829d-40abdcf040b0\") " pod="openshift-ingress/payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk"
Sep 08 16:53:21.901352 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:21.901257    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/866bb25b-8cde-40b9-829d-40abdcf040b0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk\" (UID: \"866bb25b-8cde-40b9-829d-40abdcf040b0\") " pod="openshift-ingress/payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk"
Sep 08 16:53:21.901954 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:21.901936    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/866bb25b-8cde-40b9-829d-40abdcf040b0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk\" (UID: \"866bb25b-8cde-40b9-829d-40abdcf040b0\") " pod="openshift-ingress/payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk"
Sep 08 16:53:21.917871 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:21.917828    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq7lk\" (UniqueName: \"kubernetes.io/projected/866bb25b-8cde-40b9-829d-40abdcf040b0-kube-api-access-tq7lk\") pod \"payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk\" (UID: \"866bb25b-8cde-40b9-829d-40abdcf040b0\") " pod="openshift-ingress/payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk"
Sep 08 16:53:21.944502 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:21.944481    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk"
Sep 08 16:53:22.220811 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:22.220784    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk"]
Sep 08 16:53:22.222855 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:53:22.222800    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod866bb25b_8cde_40b9_829d_40abdcf040b0.slice/crio-ecdd0d6166c66cc7e0721c47e53d6b5ce939bb2b4c2728a8946121dc0325588c WatchSource:0}: Error finding container ecdd0d6166c66cc7e0721c47e53d6b5ce939bb2b4c2728a8946121dc0325588c: Status 404 returned error can't find the container with id ecdd0d6166c66cc7e0721c47e53d6b5ce939bb2b4c2728a8946121dc0325588c
Sep 08 16:53:22.265845 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:22.265813    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:53:22.862509 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:22.862472    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk" event={"ID":"866bb25b-8cde-40b9-829d-40abdcf040b0","Type":"ContainerStarted","Data":"3ff2310c5be69982eacf633501c51b4cce25d6f384adf95fdf875e9386ff99ae"}
Sep 08 16:53:22.862888 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:22.862523    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk" event={"ID":"866bb25b-8cde-40b9-829d-40abdcf040b0","Type":"ContainerStarted","Data":"ecdd0d6166c66cc7e0721c47e53d6b5ce939bb2b4c2728a8946121dc0325588c"}
Sep 08 16:53:22.862888 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:22.862611    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk"
Sep 08 16:53:22.903639 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:22.903590    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk" podStartSLOduration=1.9035740140000001 podStartE2EDuration="1.903574014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:53:22.903033244 +0000 UTC m=+1387.066004693" watchObservedRunningTime="2026-09-08 16:53:22.903574014 +0000 UTC m=+1387.066545460"
Sep 08 16:53:33.869474 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:33.869446    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk"
Sep 08 16:53:34.090381 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:34.090351    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"]
Sep 08 16:53:35.105259 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:35.105229    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"]
Sep 08 16:53:38.945355 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:38.945324    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:53:46.954680 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:46.954651    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"]
Sep 08 16:53:46.957967 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:46.957950    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:53:46.966338 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:46.966316    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-608268\"/\"rate-test-model-608268-kserve-self-signed-certs\""
Sep 08 16:53:46.977135 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:46.977113    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"]
Sep 08 16:53:47.002229 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:47.002208    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40703dcd-7963-415f-862c-922f47f8edbc-tmp-dir\") pod \"rate-test-model-608268-kserve-6cd9886955-f7t4w\" (UID: \"40703dcd-7963-415f-862c-922f47f8edbc\") " pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:53:47.002229 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:47.002237    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40703dcd-7963-415f-862c-922f47f8edbc-home\") pod \"rate-test-model-608268-kserve-6cd9886955-f7t4w\" (UID: \"40703dcd-7963-415f-862c-922f47f8edbc\") " pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:53:47.002400 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:47.002273    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40703dcd-7963-415f-862c-922f47f8edbc-dshm\") pod \"rate-test-model-608268-kserve-6cd9886955-f7t4w\" (UID: \"40703dcd-7963-415f-862c-922f47f8edbc\") " pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:53:47.002400 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:47.002290    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40703dcd-7963-415f-862c-922f47f8edbc-model-cache\") pod \"rate-test-model-608268-kserve-6cd9886955-f7t4w\" (UID: \"40703dcd-7963-415f-862c-922f47f8edbc\") " pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:53:47.002400 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:47.002306    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40703dcd-7963-415f-862c-922f47f8edbc-tls-certs\") pod \"rate-test-model-608268-kserve-6cd9886955-f7t4w\" (UID: \"40703dcd-7963-415f-862c-922f47f8edbc\") " pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:53:47.002547 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:47.002419    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s98br\" (UniqueName: \"kubernetes.io/projected/40703dcd-7963-415f-862c-922f47f8edbc-kube-api-access-s98br\") pod \"rate-test-model-608268-kserve-6cd9886955-f7t4w\" (UID: \"40703dcd-7963-415f-862c-922f47f8edbc\") " pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:53:47.103619 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:47.103585    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s98br\" (UniqueName: \"kubernetes.io/projected/40703dcd-7963-415f-862c-922f47f8edbc-kube-api-access-s98br\") pod \"rate-test-model-608268-kserve-6cd9886955-f7t4w\" (UID: \"40703dcd-7963-415f-862c-922f47f8edbc\") " pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:53:47.103746 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:47.103657    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40703dcd-7963-415f-862c-922f47f8edbc-tmp-dir\") pod \"rate-test-model-608268-kserve-6cd9886955-f7t4w\" (UID: \"40703dcd-7963-415f-862c-922f47f8edbc\") " pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:53:47.103746 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:47.103689    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40703dcd-7963-415f-862c-922f47f8edbc-home\") pod \"rate-test-model-608268-kserve-6cd9886955-f7t4w\" (UID: \"40703dcd-7963-415f-862c-922f47f8edbc\") " pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:53:47.103746 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:47.103716    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40703dcd-7963-415f-862c-922f47f8edbc-dshm\") pod \"rate-test-model-608268-kserve-6cd9886955-f7t4w\" (UID: \"40703dcd-7963-415f-862c-922f47f8edbc\") " pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:53:47.103746 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:47.103740    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40703dcd-7963-415f-862c-922f47f8edbc-model-cache\") pod \"rate-test-model-608268-kserve-6cd9886955-f7t4w\" (UID: \"40703dcd-7963-415f-862c-922f47f8edbc\") " pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:53:47.103975 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:47.103763    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40703dcd-7963-415f-862c-922f47f8edbc-tls-certs\") pod \"rate-test-model-608268-kserve-6cd9886955-f7t4w\" (UID: \"40703dcd-7963-415f-862c-922f47f8edbc\") " pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:53:47.104097 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:47.104074    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40703dcd-7963-415f-862c-922f47f8edbc-home\") pod \"rate-test-model-608268-kserve-6cd9886955-f7t4w\" (UID: \"40703dcd-7963-415f-862c-922f47f8edbc\") " pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:53:47.104184 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:47.104163    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40703dcd-7963-415f-862c-922f47f8edbc-tmp-dir\") pod \"rate-test-model-608268-kserve-6cd9886955-f7t4w\" (UID: \"40703dcd-7963-415f-862c-922f47f8edbc\") " pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:53:47.104246 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:47.104209    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40703dcd-7963-415f-862c-922f47f8edbc-model-cache\") pod \"rate-test-model-608268-kserve-6cd9886955-f7t4w\" (UID: \"40703dcd-7963-415f-862c-922f47f8edbc\") " pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:53:47.106095 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:47.106076    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40703dcd-7963-415f-862c-922f47f8edbc-dshm\") pod \"rate-test-model-608268-kserve-6cd9886955-f7t4w\" (UID: \"40703dcd-7963-415f-862c-922f47f8edbc\") " pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:53:47.106244 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:47.106228    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40703dcd-7963-415f-862c-922f47f8edbc-tls-certs\") pod \"rate-test-model-608268-kserve-6cd9886955-f7t4w\" (UID: \"40703dcd-7963-415f-862c-922f47f8edbc\") " pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:53:47.115269 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:47.115248    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s98br\" (UniqueName: \"kubernetes.io/projected/40703dcd-7963-415f-862c-922f47f8edbc-kube-api-access-s98br\") pod \"rate-test-model-608268-kserve-6cd9886955-f7t4w\" (UID: \"40703dcd-7963-415f-862c-922f47f8edbc\") " pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:53:47.266377 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:47.266312    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:53:47.559871 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:47.559825    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"]
Sep 08 16:53:47.560409 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:53:47.560381    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40703dcd_7963_415f_862c_922f47f8edbc.slice/crio-32cd136c55b2e7fcb7e57f2c9468093064714258ebbf15accdbce15dabd65fac WatchSource:0}: Error finding container 32cd136c55b2e7fcb7e57f2c9468093064714258ebbf15accdbce15dabd65fac: Status 404 returned error can't find the container with id 32cd136c55b2e7fcb7e57f2c9468093064714258ebbf15accdbce15dabd65fac
Sep 08 16:53:47.995898 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:47.995863    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w" event={"ID":"40703dcd-7963-415f-862c-922f47f8edbc","Type":"ContainerStarted","Data":"713e7f0017b75ee16abc8c2e32c36054214999cdc3fac27304fb4461849bcf5d"}
Sep 08 16:53:47.996247 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:47.995909    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w" event={"ID":"40703dcd-7963-415f-862c-922f47f8edbc","Type":"ContainerStarted","Data":"32cd136c55b2e7fcb7e57f2c9468093064714258ebbf15accdbce15dabd65fac"}
Sep 08 16:53:48.023423 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:48.023374    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w" podStartSLOduration=2.023359284 podStartE2EDuration="2.023359284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:53:48.021141943 +0000 UTC m=+1412.184113388" watchObservedRunningTime="2026-09-08 16:53:48.023359284 +0000 UTC m=+1412.186330730"
Sep 08 16:53:49.109310 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:49.109258    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z" podUID="d3a89f53-df8c-4464-8a1b-fa6907e3db86" containerName="main" containerID="cri-o://579bcda92adf66177e5c96763082a0c8e4b78b94e0ba9134234b7a8283509298" gracePeriod=45
Sep 08 16:53:49.365099 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:49.365048    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:53:49.424446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:49.424416    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a89f53-df8c-4464-8a1b-fa6907e3db86-tmp-dir\") pod \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\" (UID: \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\") "
Sep 08 16:53:49.424446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:49.424445    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a89f53-df8c-4464-8a1b-fa6907e3db86-tls-certs\") pod \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\" (UID: \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\") "
Sep 08 16:53:49.424648 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:49.424464    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a89f53-df8c-4464-8a1b-fa6907e3db86-model-cache\") pod \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\" (UID: \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\") "
Sep 08 16:53:49.424648 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:49.424503    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2s8xx\" (UniqueName: \"kubernetes.io/projected/d3a89f53-df8c-4464-8a1b-fa6907e3db86-kube-api-access-2s8xx\") pod \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\" (UID: \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\") "
Sep 08 16:53:49.424648 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:49.424613    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3a89f53-df8c-4464-8a1b-fa6907e3db86-home\") pod \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\" (UID: \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\") "
Sep 08 16:53:49.424648 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:49.424642    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a89f53-df8c-4464-8a1b-fa6907e3db86-dshm\") pod \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\" (UID: \"d3a89f53-df8c-4464-8a1b-fa6907e3db86\") "
Sep 08 16:53:49.424866 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:49.424732    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a89f53-df8c-4464-8a1b-fa6907e3db86-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d3a89f53-df8c-4464-8a1b-fa6907e3db86" (UID: "d3a89f53-df8c-4464-8a1b-fa6907e3db86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:53:49.424866 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:49.424747    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a89f53-df8c-4464-8a1b-fa6907e3db86-model-cache" (OuterVolumeSpecName: "model-cache") pod "d3a89f53-df8c-4464-8a1b-fa6907e3db86" (UID: "d3a89f53-df8c-4464-8a1b-fa6907e3db86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:53:49.424978 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:49.424882    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a89f53-df8c-4464-8a1b-fa6907e3db86-home" (OuterVolumeSpecName: "home") pod "d3a89f53-df8c-4464-8a1b-fa6907e3db86" (UID: "d3a89f53-df8c-4464-8a1b-fa6907e3db86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:53:49.424978 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:49.424950    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a89f53-df8c-4464-8a1b-fa6907e3db86-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:53:49.424978 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:49.424968    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a89f53-df8c-4464-8a1b-fa6907e3db86-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:53:49.426716 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:49.426694    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3a89f53-df8c-4464-8a1b-fa6907e3db86-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d3a89f53-df8c-4464-8a1b-fa6907e3db86" (UID: "d3a89f53-df8c-4464-8a1b-fa6907e3db86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:53:49.426810 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:49.426724    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a89f53-df8c-4464-8a1b-fa6907e3db86-dshm" (OuterVolumeSpecName: "dshm") pod "d3a89f53-df8c-4464-8a1b-fa6907e3db86" (UID: "d3a89f53-df8c-4464-8a1b-fa6907e3db86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:53:49.426810 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:49.426739    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a89f53-df8c-4464-8a1b-fa6907e3db86-kube-api-access-2s8xx" (OuterVolumeSpecName: "kube-api-access-2s8xx") pod "d3a89f53-df8c-4464-8a1b-fa6907e3db86" (UID: "d3a89f53-df8c-4464-8a1b-fa6907e3db86"). InnerVolumeSpecName "kube-api-access-2s8xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:53:49.526295 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:49.526271    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s8xx\" (UniqueName: \"kubernetes.io/projected/d3a89f53-df8c-4464-8a1b-fa6907e3db86-kube-api-access-2s8xx\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:53:49.526295 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:49.526296    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3a89f53-df8c-4464-8a1b-fa6907e3db86-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:53:49.526439 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:49.526308    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a89f53-df8c-4464-8a1b-fa6907e3db86-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:53:49.526439 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:49.526316    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a89f53-df8c-4464-8a1b-fa6907e3db86-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:53:50.007750 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.007712    2557 generic.go:356] "Generic (PLEG): container finished" podID="d3a89f53-df8c-4464-8a1b-fa6907e3db86" containerID="579bcda92adf66177e5c96763082a0c8e4b78b94e0ba9134234b7a8283509298" exitCode=0
Sep 08 16:53:50.007930 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.007757    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"
Sep 08 16:53:50.007930 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.007797    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z" event={"ID":"d3a89f53-df8c-4464-8a1b-fa6907e3db86","Type":"ContainerDied","Data":"579bcda92adf66177e5c96763082a0c8e4b78b94e0ba9134234b7a8283509298"}
Sep 08 16:53:50.007930 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.007851    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z" event={"ID":"d3a89f53-df8c-4464-8a1b-fa6907e3db86","Type":"ContainerDied","Data":"ba168f49cf9c91af816dba513f159cc7ff93a352d56acfd91dde2180387f90d1"}
Sep 08 16:53:50.007930 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.007870    2557 scope.go:117] "RemoveContainer" containerID="579bcda92adf66177e5c96763082a0c8e4b78b94e0ba9134234b7a8283509298"
Sep 08 16:53:50.017109 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.017083    2557 scope.go:117] "RemoveContainer" containerID="579bcda92adf66177e5c96763082a0c8e4b78b94e0ba9134234b7a8283509298"
Sep 08 16:53:50.017381 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:53:50.017364    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"579bcda92adf66177e5c96763082a0c8e4b78b94e0ba9134234b7a8283509298\": container with ID starting with 579bcda92adf66177e5c96763082a0c8e4b78b94e0ba9134234b7a8283509298 not found: ID does not exist" containerID="579bcda92adf66177e5c96763082a0c8e4b78b94e0ba9134234b7a8283509298"
Sep 08 16:53:50.017438 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.017392    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"579bcda92adf66177e5c96763082a0c8e4b78b94e0ba9134234b7a8283509298"} err="failed to get container status \"579bcda92adf66177e5c96763082a0c8e4b78b94e0ba9134234b7a8283509298\": rpc error: code = NotFound desc = could not find container \"579bcda92adf66177e5c96763082a0c8e4b78b94e0ba9134234b7a8283509298\": container with ID starting with 579bcda92adf66177e5c96763082a0c8e4b78b94e0ba9134234b7a8283509298 not found: ID does not exist"
Sep 08 16:53:50.040887 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.040857    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"]
Sep 08 16:53:50.044684 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.044661    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-608268/sub-test-model-608268-kserve-5c74cff87f-rtc6z"]
Sep 08 16:53:50.124185 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.124146    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr" podUID="9b0a3597-fb4e-4b54-b3d3-60c194c07caf" containerName="main" containerID="cri-o://a4c553df45f490285fdbe05a0464e516d715eddb110e32ae886da12932f686de" gracePeriod=45
Sep 08 16:53:50.346668 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.346636    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3a89f53-df8c-4464-8a1b-fa6907e3db86" path="/var/lib/kubelet/pods/d3a89f53-df8c-4464-8a1b-fa6907e3db86/volumes"
Sep 08 16:53:50.375914 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.375887    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:53:50.433769 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.433742    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-tmp-dir\") pod \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\" (UID: \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\") "
Sep 08 16:53:50.433934 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.433778    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-home\") pod \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\" (UID: \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\") "
Sep 08 16:53:50.433934 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.433818    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-tls-certs\") pod \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\" (UID: \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\") "
Sep 08 16:53:50.433934 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.433858    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgwkh\" (UniqueName: \"kubernetes.io/projected/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-kube-api-access-zgwkh\") pod \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\" (UID: \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\") "
Sep 08 16:53:50.434115 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.433948    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-dshm\") pod \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\" (UID: \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\") "
Sep 08 16:53:50.434115 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.433978    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-model-cache\") pod \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\" (UID: \"9b0a3597-fb4e-4b54-b3d3-60c194c07caf\") "
Sep 08 16:53:50.434115 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.434075    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-home" (OuterVolumeSpecName: "home") pod "9b0a3597-fb4e-4b54-b3d3-60c194c07caf" (UID: "9b0a3597-fb4e-4b54-b3d3-60c194c07caf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:53:50.434115 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.434097    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b0a3597-fb4e-4b54-b3d3-60c194c07caf" (UID: "9b0a3597-fb4e-4b54-b3d3-60c194c07caf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:53:50.434330 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.434307    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b0a3597-fb4e-4b54-b3d3-60c194c07caf" (UID: "9b0a3597-fb4e-4b54-b3d3-60c194c07caf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:53:50.434387 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.434334    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:53:50.434387 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.434358    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:53:50.436172 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.436151    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-dshm" (OuterVolumeSpecName: "dshm") pod "9b0a3597-fb4e-4b54-b3d3-60c194c07caf" (UID: "9b0a3597-fb4e-4b54-b3d3-60c194c07caf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:53:50.436172 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.436165    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-kube-api-access-zgwkh" (OuterVolumeSpecName: "kube-api-access-zgwkh") pod "9b0a3597-fb4e-4b54-b3d3-60c194c07caf" (UID: "9b0a3597-fb4e-4b54-b3d3-60c194c07caf"). InnerVolumeSpecName "kube-api-access-zgwkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:53:50.436309 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.436227    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b0a3597-fb4e-4b54-b3d3-60c194c07caf" (UID: "9b0a3597-fb4e-4b54-b3d3-60c194c07caf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:53:50.535152 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.535128    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:53:50.535152 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.535151    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:53:50.535152 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.535160    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:53:50.535349 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:50.535169    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgwkh\" (UniqueName: \"kubernetes.io/projected/9b0a3597-fb4e-4b54-b3d3-60c194c07caf-kube-api-access-zgwkh\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:53:51.015325 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:51.015294    2557 generic.go:356] "Generic (PLEG): container finished" podID="9b0a3597-fb4e-4b54-b3d3-60c194c07caf" containerID="a4c553df45f490285fdbe05a0464e516d715eddb110e32ae886da12932f686de" exitCode=0
Sep 08 16:53:51.015471 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:51.015337    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"
Sep 08 16:53:51.015471 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:51.015346    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr" event={"ID":"9b0a3597-fb4e-4b54-b3d3-60c194c07caf","Type":"ContainerDied","Data":"a4c553df45f490285fdbe05a0464e516d715eddb110e32ae886da12932f686de"}
Sep 08 16:53:51.015471 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:51.015376    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr" event={"ID":"9b0a3597-fb4e-4b54-b3d3-60c194c07caf","Type":"ContainerDied","Data":"70c1211a246acfe231f97743dc7fb0868e8cb7abcc4dcaebf594ed9f1f35e4a1"}
Sep 08 16:53:51.015471 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:51.015400    2557 scope.go:117] "RemoveContainer" containerID="a4c553df45f490285fdbe05a0464e516d715eddb110e32ae886da12932f686de"
Sep 08 16:53:51.028811 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:51.028795    2557 scope.go:117] "RemoveContainer" containerID="a4c553df45f490285fdbe05a0464e516d715eddb110e32ae886da12932f686de"
Sep 08 16:53:51.029078 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:53:51.029060    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4c553df45f490285fdbe05a0464e516d715eddb110e32ae886da12932f686de\": container with ID starting with a4c553df45f490285fdbe05a0464e516d715eddb110e32ae886da12932f686de not found: ID does not exist" containerID="a4c553df45f490285fdbe05a0464e516d715eddb110e32ae886da12932f686de"
Sep 08 16:53:51.029121 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:51.029085    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c553df45f490285fdbe05a0464e516d715eddb110e32ae886da12932f686de"} err="failed to get container status \"a4c553df45f490285fdbe05a0464e516d715eddb110e32ae886da12932f686de\": rpc error: code = NotFound desc = could not find container \"a4c553df45f490285fdbe05a0464e516d715eddb110e32ae886da12932f686de\": container with ID starting with a4c553df45f490285fdbe05a0464e516d715eddb110e32ae886da12932f686de not found: ID does not exist"
Sep 08 16:53:51.043635 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:51.043609    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"]
Sep 08 16:53:51.047706 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:51.047685    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6b15ca/sub-test-model-6b15ca-kserve-578f88b859-txdzr"]
Sep 08 16:53:52.345397 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:52.345361    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b0a3597-fb4e-4b54-b3d3-60c194c07caf" path="/var/lib/kubelet/pods/9b0a3597-fb4e-4b54-b3d3-60c194c07caf/volumes"
Sep 08 16:53:53.180543 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:53.180511    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:53:55.654253 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.654218    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n"]
Sep 08 16:53:55.654774 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.654755    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b0a3597-fb4e-4b54-b3d3-60c194c07caf" containerName="main"
Sep 08 16:53:55.654887 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.654779    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b0a3597-fb4e-4b54-b3d3-60c194c07caf" containerName="main"
Sep 08 16:53:55.654887 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.654795    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3a89f53-df8c-4464-8a1b-fa6907e3db86" containerName="main"
Sep 08 16:53:55.654887 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.654803    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3a89f53-df8c-4464-8a1b-fa6907e3db86" containerName="main"
Sep 08 16:53:55.655059 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.654925    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b0a3597-fb4e-4b54-b3d3-60c194c07caf" containerName="main"
Sep 08 16:53:55.655059 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.654942    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3a89f53-df8c-4464-8a1b-fa6907e3db86" containerName="main"
Sep 08 16:53:55.659300 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.659280    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n"
Sep 08 16:53:55.664311 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.664292    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-7818c2-dockercfg-kk57m\""
Sep 08 16:53:55.664640 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.664624    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-7818c2\""
Sep 08 16:53:55.678102 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.678083    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n"]
Sep 08 16:53:55.740325 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.740295    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd"]
Sep 08 16:53:55.744006 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.743990    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd"
Sep 08 16:53:55.760468 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.760444    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd"]
Sep 08 16:53:55.774810 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.774781    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f839060a-9e2a-4042-adaa-04915065489f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd\" (UID: \"f839060a-9e2a-4042-adaa-04915065489f\") " pod="openshift-ingress/payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd"
Sep 08 16:53:55.774941 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.774854    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsp4r\" (UniqueName: \"kubernetes.io/projected/f839060a-9e2a-4042-adaa-04915065489f-kube-api-access-lsp4r\") pod \"payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd\" (UID: \"f839060a-9e2a-4042-adaa-04915065489f\") " pod="openshift-ingress/payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd"
Sep 08 16:53:55.774941 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.774927    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b87e294-ec93-4f11-867b-04e176f28a03-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n\" (UID: \"5b87e294-ec93-4f11-867b-04e176f28a03\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n"
Sep 08 16:53:55.775057 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.774996    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwlc8\" (UniqueName: \"kubernetes.io/projected/5b87e294-ec93-4f11-867b-04e176f28a03-kube-api-access-mwlc8\") pod \"payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n\" (UID: \"5b87e294-ec93-4f11-867b-04e176f28a03\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n"
Sep 08 16:53:55.875780 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.875748    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsp4r\" (UniqueName: \"kubernetes.io/projected/f839060a-9e2a-4042-adaa-04915065489f-kube-api-access-lsp4r\") pod \"payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd\" (UID: \"f839060a-9e2a-4042-adaa-04915065489f\") " pod="openshift-ingress/payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd"
Sep 08 16:53:55.875949 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.875789    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b87e294-ec93-4f11-867b-04e176f28a03-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n\" (UID: \"5b87e294-ec93-4f11-867b-04e176f28a03\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n"
Sep 08 16:53:55.875949 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.875826    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwlc8\" (UniqueName: \"kubernetes.io/projected/5b87e294-ec93-4f11-867b-04e176f28a03-kube-api-access-mwlc8\") pod \"payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n\" (UID: \"5b87e294-ec93-4f11-867b-04e176f28a03\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n"
Sep 08 16:53:55.875949 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.875876    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f839060a-9e2a-4042-adaa-04915065489f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd\" (UID: \"f839060a-9e2a-4042-adaa-04915065489f\") " pod="openshift-ingress/payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd"
Sep 08 16:53:55.876498 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.876470    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f839060a-9e2a-4042-adaa-04915065489f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd\" (UID: \"f839060a-9e2a-4042-adaa-04915065489f\") " pod="openshift-ingress/payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd"
Sep 08 16:53:55.876953 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.876928    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b87e294-ec93-4f11-867b-04e176f28a03-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n\" (UID: \"5b87e294-ec93-4f11-867b-04e176f28a03\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n"
Sep 08 16:53:55.887609 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.887588    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsp4r\" (UniqueName: \"kubernetes.io/projected/f839060a-9e2a-4042-adaa-04915065489f-kube-api-access-lsp4r\") pod \"payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd\" (UID: \"f839060a-9e2a-4042-adaa-04915065489f\") " pod="openshift-ingress/payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd"
Sep 08 16:53:55.887772 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.887753    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwlc8\" (UniqueName: \"kubernetes.io/projected/5b87e294-ec93-4f11-867b-04e176f28a03-kube-api-access-mwlc8\") pod \"payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n\" (UID: \"5b87e294-ec93-4f11-867b-04e176f28a03\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n"
Sep 08 16:53:55.968586 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:55.968557    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n"
Sep 08 16:53:56.053859 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:56.053818    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd"
Sep 08 16:53:56.249601 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:56.249581    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n"]
Sep 08 16:53:56.251620 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:53:56.251596    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b87e294_ec93_4f11_867b_04e176f28a03.slice/crio-b66759344a96258a04b20c95a54be25050158ab4f6941f4e50525e6a1cabfb9c WatchSource:0}: Error finding container b66759344a96258a04b20c95a54be25050158ab4f6941f4e50525e6a1cabfb9c: Status 404 returned error can't find the container with id b66759344a96258a04b20c95a54be25050158ab4f6941f4e50525e6a1cabfb9c
Sep 08 16:53:56.352620 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:56.352598    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd"]
Sep 08 16:53:56.353866 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:53:56.353824    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf839060a_9e2a_4042_adaa_04915065489f.slice/crio-ecf773c6dac79c8ed6c8dc59dc4504deca41b6d7100fd202fb1b5dfe626d5441 WatchSource:0}: Error finding container ecf773c6dac79c8ed6c8dc59dc4504deca41b6d7100fd202fb1b5dfe626d5441: Status 404 returned error can't find the container with id ecf773c6dac79c8ed6c8dc59dc4504deca41b6d7100fd202fb1b5dfe626d5441
Sep 08 16:53:57.048243 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:57.048203    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd" event={"ID":"f839060a-9e2a-4042-adaa-04915065489f","Type":"ContainerStarted","Data":"b14f5344d69e17cae6a52058e85fe477a736fd01f76baaab754514549ac45fe0"}
Sep 08 16:53:57.048677 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:57.048265    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd" event={"ID":"f839060a-9e2a-4042-adaa-04915065489f","Type":"ContainerStarted","Data":"ecf773c6dac79c8ed6c8dc59dc4504deca41b6d7100fd202fb1b5dfe626d5441"}
Sep 08 16:53:57.048677 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:57.048395    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd"
Sep 08 16:53:57.049760 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:57.049730    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n" event={"ID":"5b87e294-ec93-4f11-867b-04e176f28a03","Type":"ContainerStarted","Data":"1195abf428bcf93e894c57d7492ba6e107928ad972577edf19b9c7b635f36d37"}
Sep 08 16:53:57.049877 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:57.049770    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n" event={"ID":"5b87e294-ec93-4f11-867b-04e176f28a03","Type":"ContainerStarted","Data":"b66759344a96258a04b20c95a54be25050158ab4f6941f4e50525e6a1cabfb9c"}
Sep 08 16:53:57.049877 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:57.049797    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n"
Sep 08 16:53:57.077505 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:57.077459    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd" podStartSLOduration=2.077446878 podStartE2EDuration="2.077446878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:53:57.075672378 +0000 UTC m=+1421.238643827" watchObservedRunningTime="2026-09-08 16:53:57.077446878 +0000 UTC m=+1421.240418483"
Sep 08 16:53:57.121294 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:57.121250    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n" podStartSLOduration=2.121237931 podStartE2EDuration="2.121237931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:53:57.120468167 +0000 UTC m=+1421.283439628" watchObservedRunningTime="2026-09-08 16:53:57.121237931 +0000 UTC m=+1421.284209409"
Sep 08 16:53:57.267381 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:57.267341    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:53:57.267552 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:57.267393    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:53:57.280874 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:57.280846    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:53:58.065614 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:53:58.065588    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:54:06.739559 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.739522    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"]
Sep 08 16:54:06.743384 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.743362    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:54:06.748403 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.748375    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6b15ca\"/\"kube-root-ca.crt\""
Sep 08 16:54:06.749702 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.749676    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6b15ca\"/\"rate-test-model-6b15ca-kserve-self-signed-certs\""
Sep 08 16:54:06.749945 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.749730    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6b15ca\"/\"default-dockercfg-c8vzm\""
Sep 08 16:54:06.750049 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.749789    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6b15ca\"/\"openshift-service-ca.crt\""
Sep 08 16:54:06.761798 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.761769    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"]
Sep 08 16:54:06.879093 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.879006    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac553768-4a41-43c5-aef7-0e9ae8cb726d-home\") pod \"rate-test-model-6b15ca-kserve-bb79b8555-wqppd\" (UID: \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:54:06.879093 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.879070    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac553768-4a41-43c5-aef7-0e9ae8cb726d-tls-certs\") pod \"rate-test-model-6b15ca-kserve-bb79b8555-wqppd\" (UID: \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:54:06.879342 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.879115    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac553768-4a41-43c5-aef7-0e9ae8cb726d-dshm\") pod \"rate-test-model-6b15ca-kserve-bb79b8555-wqppd\" (UID: \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:54:06.879342 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.879138    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4knkw\" (UniqueName: \"kubernetes.io/projected/ac553768-4a41-43c5-aef7-0e9ae8cb726d-kube-api-access-4knkw\") pod \"rate-test-model-6b15ca-kserve-bb79b8555-wqppd\" (UID: \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:54:06.879342 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.879197    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac553768-4a41-43c5-aef7-0e9ae8cb726d-model-cache\") pod \"rate-test-model-6b15ca-kserve-bb79b8555-wqppd\" (UID: \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:54:06.879342 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.879267    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac553768-4a41-43c5-aef7-0e9ae8cb726d-tmp-dir\") pod \"rate-test-model-6b15ca-kserve-bb79b8555-wqppd\" (UID: \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:54:06.980579 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.980535    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac553768-4a41-43c5-aef7-0e9ae8cb726d-home\") pod \"rate-test-model-6b15ca-kserve-bb79b8555-wqppd\" (UID: \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:54:06.980788 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.980597    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac553768-4a41-43c5-aef7-0e9ae8cb726d-tls-certs\") pod \"rate-test-model-6b15ca-kserve-bb79b8555-wqppd\" (UID: \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:54:06.980788 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.980653    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac553768-4a41-43c5-aef7-0e9ae8cb726d-dshm\") pod \"rate-test-model-6b15ca-kserve-bb79b8555-wqppd\" (UID: \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:54:06.980788 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.980678    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4knkw\" (UniqueName: \"kubernetes.io/projected/ac553768-4a41-43c5-aef7-0e9ae8cb726d-kube-api-access-4knkw\") pod \"rate-test-model-6b15ca-kserve-bb79b8555-wqppd\" (UID: \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:54:06.980788 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.980706    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac553768-4a41-43c5-aef7-0e9ae8cb726d-model-cache\") pod \"rate-test-model-6b15ca-kserve-bb79b8555-wqppd\" (UID: \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:54:06.980788 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.980779    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac553768-4a41-43c5-aef7-0e9ae8cb726d-tmp-dir\") pod \"rate-test-model-6b15ca-kserve-bb79b8555-wqppd\" (UID: \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:54:06.981129 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.981107    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac553768-4a41-43c5-aef7-0e9ae8cb726d-home\") pod \"rate-test-model-6b15ca-kserve-bb79b8555-wqppd\" (UID: \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:54:06.981129 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.981119    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac553768-4a41-43c5-aef7-0e9ae8cb726d-tmp-dir\") pod \"rate-test-model-6b15ca-kserve-bb79b8555-wqppd\" (UID: \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:54:06.981224 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.981140    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac553768-4a41-43c5-aef7-0e9ae8cb726d-model-cache\") pod \"rate-test-model-6b15ca-kserve-bb79b8555-wqppd\" (UID: \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:54:06.983058 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.983038    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac553768-4a41-43c5-aef7-0e9ae8cb726d-dshm\") pod \"rate-test-model-6b15ca-kserve-bb79b8555-wqppd\" (UID: \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:54:06.983381 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.983363    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac553768-4a41-43c5-aef7-0e9ae8cb726d-tls-certs\") pod \"rate-test-model-6b15ca-kserve-bb79b8555-wqppd\" (UID: \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:54:06.991965 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:06.991902    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4knkw\" (UniqueName: \"kubernetes.io/projected/ac553768-4a41-43c5-aef7-0e9ae8cb726d-kube-api-access-4knkw\") pod \"rate-test-model-6b15ca-kserve-bb79b8555-wqppd\" (UID: \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\") " pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:54:07.054152 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:07.054113    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:54:07.340231 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:07.340194    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"]
Sep 08 16:54:07.342178 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:54:07.342145    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac553768_4a41_43c5_aef7_0e9ae8cb726d.slice/crio-880cbf3ce944d69c78ea7faceca232c6cf409e247faf2704ee2bd21a7b40e473 WatchSource:0}: Error finding container 880cbf3ce944d69c78ea7faceca232c6cf409e247faf2704ee2bd21a7b40e473: Status 404 returned error can't find the container with id 880cbf3ce944d69c78ea7faceca232c6cf409e247faf2704ee2bd21a7b40e473
Sep 08 16:54:08.056718 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:08.056687    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd"
Sep 08 16:54:08.057202 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:08.056757    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n"
Sep 08 16:54:08.114407 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:08.114362    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd" event={"ID":"ac553768-4a41-43c5-aef7-0e9ae8cb726d","Type":"ContainerStarted","Data":"77e83dd3919e8009fc50260bad0f2bc8c59a6a061321df05e4ff71e8dfd561c4"}
Sep 08 16:54:08.114407 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:08.114420    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd" event={"ID":"ac553768-4a41-43c5-aef7-0e9ae8cb726d","Type":"ContainerStarted","Data":"880cbf3ce944d69c78ea7faceca232c6cf409e247faf2704ee2bd21a7b40e473"}
Sep 08 16:54:08.144946 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:08.144893    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd" podStartSLOduration=2.144872375 podStartE2EDuration="2.144872375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:54:08.141007323 +0000 UTC m=+1432.303978785" watchObservedRunningTime="2026-09-08 16:54:08.144872375 +0000 UTC m=+1432.307843823"
Sep 08 16:54:09.147436 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:09.147392    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:54:16.603503 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:16.603468    2557 scope.go:117] "RemoveContainer" containerID="722e48cba9789f9d213ac31f97b7cd338d16f2b08a766e5a3c57ff1c6b560352"
Sep 08 16:54:16.612362 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:16.612339    2557 scope.go:117] "RemoveContainer" containerID="4c67d9354f00859d58823966571976366ce61a3d8e9715217edfdd35f78a34aa"
Sep 08 16:54:17.054462 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:17.054422    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:54:17.054656 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:17.054482    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:54:17.067624 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:17.067590    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:54:17.176666 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:17.176634    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:54:24.362129 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:24.362098    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:54:26.627544 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:26.627514    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-246729-4k8b5"]
Sep 08 16:54:26.633011 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:26.632988    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-246729-4k8b5"
Sep 08 16:54:26.635529 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:26.635505    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-886jf\""
Sep 08 16:54:26.636006 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:26.635989    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:54:26.652757 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:26.652720    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-246729-4k8b5"]
Sep 08 16:54:26.765526 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:26.765482    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d160052-55eb-446d-8da1-53da67eb53b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-246729-4k8b5\" (UID: \"6d160052-55eb-446d-8da1-53da67eb53b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-246729-4k8b5"
Sep 08 16:54:26.866918 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:26.866881    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d160052-55eb-446d-8da1-53da67eb53b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-246729-4k8b5\" (UID: \"6d160052-55eb-446d-8da1-53da67eb53b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-246729-4k8b5"
Sep 08 16:54:26.867482 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:26.867461    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d160052-55eb-446d-8da1-53da67eb53b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-246729-4k8b5\" (UID: \"6d160052-55eb-446d-8da1-53da67eb53b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-246729-4k8b5"
Sep 08 16:54:26.944583 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:26.944550    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-246729-4k8b5"
Sep 08 16:54:27.225200 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:27.225176    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-246729-4k8b5"]
Sep 08 16:54:28.224607 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:28.224566    2557 generic.go:356] "Generic (PLEG): container finished" podID="6d160052-55eb-446d-8da1-53da67eb53b5" containerID="4babb6ab70cf3ae0b53abafe25fb197720c9c47257a34f57085201d73c352470" exitCode=0
Sep 08 16:54:28.225025 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:28.224667    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-246729-4k8b5" event={"ID":"6d160052-55eb-446d-8da1-53da67eb53b5","Type":"ContainerDied","Data":"4babb6ab70cf3ae0b53abafe25fb197720c9c47257a34f57085201d73c352470"}
Sep 08 16:54:28.225025 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:28.224723    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-246729-4k8b5" event={"ID":"6d160052-55eb-446d-8da1-53da67eb53b5","Type":"ContainerStarted","Data":"57bf00e67e50fe08fa7f0c346070ccd58a526da393c8045ba76e5117f0f2c10d"}
Sep 08 16:54:29.367814 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:29.367789    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-246729-4k8b5"
Sep 08 16:54:29.490515 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:29.490415    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d160052-55eb-446d-8da1-53da67eb53b5-maas-api-service-ca\") pod \"6d160052-55eb-446d-8da1-53da67eb53b5\" (UID: \"6d160052-55eb-446d-8da1-53da67eb53b5\") "
Sep 08 16:54:29.490814 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:29.490789    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d160052-55eb-446d-8da1-53da67eb53b5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d160052-55eb-446d-8da1-53da67eb53b5" (UID: "6d160052-55eb-446d-8da1-53da67eb53b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:54:29.591424 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:29.591382    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d160052-55eb-446d-8da1-53da67eb53b5-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:54:30.243145 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:30.243109    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-246729-4k8b5" event={"ID":"6d160052-55eb-446d-8da1-53da67eb53b5","Type":"ContainerDied","Data":"57bf00e67e50fe08fa7f0c346070ccd58a526da393c8045ba76e5117f0f2c10d"}
Sep 08 16:54:30.243373 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:30.243185    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57bf00e67e50fe08fa7f0c346070ccd58a526da393c8045ba76e5117f0f2c10d"
Sep 08 16:54:30.243373 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:30.243122    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-246729-4k8b5"
Sep 08 16:54:42.483226 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:42.483192    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk"]
Sep 08 16:54:42.483711 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:42.483417    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk" podUID="866bb25b-8cde-40b9-829d-40abdcf040b0" containerName="payload-processing" containerID="cri-o://3ff2310c5be69982eacf633501c51b4cce25d6f384adf95fdf875e9386ff99ae" gracePeriod=30
Sep 08 16:54:42.744356 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:42.744278    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk"
Sep 08 16:54:42.774410 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:42.774374    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-246729-4k8b5"]
Sep 08 16:54:42.777237 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:42.777210    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-246729-4k8b5"]
Sep 08 16:54:42.806890 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:42.806857    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq7lk\" (UniqueName: \"kubernetes.io/projected/866bb25b-8cde-40b9-829d-40abdcf040b0-kube-api-access-tq7lk\") pod \"866bb25b-8cde-40b9-829d-40abdcf040b0\" (UID: \"866bb25b-8cde-40b9-829d-40abdcf040b0\") "
Sep 08 16:54:42.807065 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:42.806911    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/866bb25b-8cde-40b9-829d-40abdcf040b0-plugins-config-volume\") pod \"866bb25b-8cde-40b9-829d-40abdcf040b0\" (UID: \"866bb25b-8cde-40b9-829d-40abdcf040b0\") "
Sep 08 16:54:42.807296 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:42.807272    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/866bb25b-8cde-40b9-829d-40abdcf040b0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "866bb25b-8cde-40b9-829d-40abdcf040b0" (UID: "866bb25b-8cde-40b9-829d-40abdcf040b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:54:42.809390 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:42.809355    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/866bb25b-8cde-40b9-829d-40abdcf040b0-kube-api-access-tq7lk" (OuterVolumeSpecName: "kube-api-access-tq7lk") pod "866bb25b-8cde-40b9-829d-40abdcf040b0" (UID: "866bb25b-8cde-40b9-829d-40abdcf040b0"). InnerVolumeSpecName "kube-api-access-tq7lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:54:42.907792 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:42.907746    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/866bb25b-8cde-40b9-829d-40abdcf040b0-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:54:42.907792 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:42.907783    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq7lk\" (UniqueName: \"kubernetes.io/projected/866bb25b-8cde-40b9-829d-40abdcf040b0-kube-api-access-tq7lk\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:54:43.312981 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:43.312942    2557 generic.go:356] "Generic (PLEG): container finished" podID="866bb25b-8cde-40b9-829d-40abdcf040b0" containerID="3ff2310c5be69982eacf633501c51b4cce25d6f384adf95fdf875e9386ff99ae" exitCode=0
Sep 08 16:54:43.313202 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:43.313013    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk"
Sep 08 16:54:43.313202 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:43.313035    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk" event={"ID":"866bb25b-8cde-40b9-829d-40abdcf040b0","Type":"ContainerDied","Data":"3ff2310c5be69982eacf633501c51b4cce25d6f384adf95fdf875e9386ff99ae"}
Sep 08 16:54:43.313202 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:43.313131    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk" event={"ID":"866bb25b-8cde-40b9-829d-40abdcf040b0","Type":"ContainerDied","Data":"ecdd0d6166c66cc7e0721c47e53d6b5ce939bb2b4c2728a8946121dc0325588c"}
Sep 08 16:54:43.313202 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:43.313160    2557 scope.go:117] "RemoveContainer" containerID="3ff2310c5be69982eacf633501c51b4cce25d6f384adf95fdf875e9386ff99ae"
Sep 08 16:54:43.323042 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:43.323023    2557 scope.go:117] "RemoveContainer" containerID="3ff2310c5be69982eacf633501c51b4cce25d6f384adf95fdf875e9386ff99ae"
Sep 08 16:54:43.323390 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:54:43.323371    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ff2310c5be69982eacf633501c51b4cce25d6f384adf95fdf875e9386ff99ae\": container with ID starting with 3ff2310c5be69982eacf633501c51b4cce25d6f384adf95fdf875e9386ff99ae not found: ID does not exist" containerID="3ff2310c5be69982eacf633501c51b4cce25d6f384adf95fdf875e9386ff99ae"
Sep 08 16:54:43.323454 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:43.323404    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ff2310c5be69982eacf633501c51b4cce25d6f384adf95fdf875e9386ff99ae"} err="failed to get container status \"3ff2310c5be69982eacf633501c51b4cce25d6f384adf95fdf875e9386ff99ae\": rpc error: code = NotFound desc = could not find container \"3ff2310c5be69982eacf633501c51b4cce25d6f384adf95fdf875e9386ff99ae\": container with ID starting with 3ff2310c5be69982eacf633501c51b4cce25d6f384adf95fdf875e9386ff99ae not found: ID does not exist"
Sep 08 16:54:43.338532 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:43.338498    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk"]
Sep 08 16:54:43.342966 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:43.342934    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-246729-8f444dcf9-5j9vk"]
Sep 08 16:54:44.346627 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:44.346587    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d160052-55eb-446d-8da1-53da67eb53b5" path="/var/lib/kubelet/pods/6d160052-55eb-446d-8da1-53da67eb53b5/volumes"
Sep 08 16:54:44.347047 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:44.346995    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="866bb25b-8cde-40b9-829d-40abdcf040b0" path="/var/lib/kubelet/pods/866bb25b-8cde-40b9-829d-40abdcf040b0/volumes"
Sep 08 16:54:49.754535 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:49.754498    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:54:52.210469 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:52.210433    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7818c2-9bm74"]
Sep 08 16:54:52.210907 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:52.210889    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="866bb25b-8cde-40b9-829d-40abdcf040b0" containerName="payload-processing"
Sep 08 16:54:52.210949 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:52.210910    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="866bb25b-8cde-40b9-829d-40abdcf040b0" containerName="payload-processing"
Sep 08 16:54:52.210949 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:52.210925    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d160052-55eb-446d-8da1-53da67eb53b5" containerName="revoke-keys"
Sep 08 16:54:52.210949 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:52.210930    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d160052-55eb-446d-8da1-53da67eb53b5" containerName="revoke-keys"
Sep 08 16:54:52.211051 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:52.211002    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="866bb25b-8cde-40b9-829d-40abdcf040b0" containerName="payload-processing"
Sep 08 16:54:52.211051 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:52.211014    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d160052-55eb-446d-8da1-53da67eb53b5" containerName="revoke-keys"
Sep 08 16:54:52.214080 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:52.214062    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7818c2-9bm74"
Sep 08 16:54:52.217141 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:52.217114    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-886jf\""
Sep 08 16:54:52.217468 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:52.217452    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:54:52.240516 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:52.240479    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7818c2-9bm74"]
Sep 08 16:54:52.295613 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:52.295573    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e204bded-d118-4781-9275-bf23a80d817b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7818c2-9bm74\" (UID: \"e204bded-d118-4781-9275-bf23a80d817b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7818c2-9bm74"
Sep 08 16:54:52.396886 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:52.396821    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e204bded-d118-4781-9275-bf23a80d817b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7818c2-9bm74\" (UID: \"e204bded-d118-4781-9275-bf23a80d817b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7818c2-9bm74"
Sep 08 16:54:52.397467 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:52.397444    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e204bded-d118-4781-9275-bf23a80d817b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7818c2-9bm74\" (UID: \"e204bded-d118-4781-9275-bf23a80d817b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7818c2-9bm74"
Sep 08 16:54:52.524465 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:52.524369    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7818c2-9bm74"
Sep 08 16:54:52.837739 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:52.837645    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7818c2-9bm74"]
Sep 08 16:54:52.840989 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:54:52.840955    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode204bded_d118_4781_9275_bf23a80d817b.slice/crio-a9f7d55813214c2c873133c2d6956bed2e6dbfd7dbb757e24aa8ade37a2df6f2 WatchSource:0}: Error finding container a9f7d55813214c2c873133c2d6956bed2e6dbfd7dbb757e24aa8ade37a2df6f2: Status 404 returned error can't find the container with id a9f7d55813214c2c873133c2d6956bed2e6dbfd7dbb757e24aa8ade37a2df6f2
Sep 08 16:54:53.369239 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:53.369202    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7818c2-9bm74" event={"ID":"e204bded-d118-4781-9275-bf23a80d817b","Type":"ContainerStarted","Data":"84478ccb29a9920f310ba2cdf1d5413b71f7c6260df2df2f3766540e748d1091"}
Sep 08 16:54:53.369648 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:53.369262    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7818c2-9bm74" event={"ID":"e204bded-d118-4781-9275-bf23a80d817b","Type":"ContainerStarted","Data":"a9f7d55813214c2c873133c2d6956bed2e6dbfd7dbb757e24aa8ade37a2df6f2"}
Sep 08 16:54:53.394378 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:53.394326    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7818c2-9bm74" podStartSLOduration=1.394310019 podStartE2EDuration="1.394310019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:54:53.393437731 +0000 UTC m=+1477.556409179" watchObservedRunningTime="2026-09-08 16:54:53.394310019 +0000 UTC m=+1477.557281485"
Sep 08 16:54:54.376049 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:54.376010    2557 generic.go:356] "Generic (PLEG): container finished" podID="e204bded-d118-4781-9275-bf23a80d817b" containerID="84478ccb29a9920f310ba2cdf1d5413b71f7c6260df2df2f3766540e748d1091" exitCode=0
Sep 08 16:54:54.376473 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:54.376088    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7818c2-9bm74" event={"ID":"e204bded-d118-4781-9275-bf23a80d817b","Type":"ContainerDied","Data":"84478ccb29a9920f310ba2cdf1d5413b71f7c6260df2df2f3766540e748d1091"}
Sep 08 16:54:55.520524 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:55.520497    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7818c2-9bm74"
Sep 08 16:54:55.625745 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:55.625642    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e204bded-d118-4781-9275-bf23a80d817b-maas-api-service-ca\") pod \"e204bded-d118-4781-9275-bf23a80d817b\" (UID: \"e204bded-d118-4781-9275-bf23a80d817b\") "
Sep 08 16:54:55.626042 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:55.626017    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e204bded-d118-4781-9275-bf23a80d817b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e204bded-d118-4781-9275-bf23a80d817b" (UID: "e204bded-d118-4781-9275-bf23a80d817b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:54:55.726562 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:55.726528    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e204bded-d118-4781-9275-bf23a80d817b-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:54:56.388802 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:56.388771    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7818c2-9bm74"
Sep 08 16:54:56.389017 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:56.388771    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7818c2-9bm74" event={"ID":"e204bded-d118-4781-9275-bf23a80d817b","Type":"ContainerDied","Data":"a9f7d55813214c2c873133c2d6956bed2e6dbfd7dbb757e24aa8ade37a2df6f2"}
Sep 08 16:54:56.389017 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:54:56.388894    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9f7d55813214c2c873133c2d6956bed2e6dbfd7dbb757e24aa8ade37a2df6f2"
Sep 08 16:55:08.149735 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.149698    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd"]
Sep 08 16:55:08.150286 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.149991    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd" podUID="f839060a-9e2a-4042-adaa-04915065489f" containerName="payload-processing" containerID="cri-o://b14f5344d69e17cae6a52058e85fe477a736fd01f76baaab754514549ac45fe0" gracePeriod=30
Sep 08 16:55:08.163244 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.163204    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n"]
Sep 08 16:55:08.163482 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.163455    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n" podUID="5b87e294-ec93-4f11-867b-04e176f28a03" containerName="payload-pre-processing" containerID="cri-o://1195abf428bcf93e894c57d7492ba6e107928ad972577edf19b9c7b635f36d37" gracePeriod=30
Sep 08 16:55:08.450214 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.450187    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7818c2-9bm74"]
Sep 08 16:55:08.453959 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.453878    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n" event={"ID":"5b87e294-ec93-4f11-867b-04e176f28a03","Type":"ContainerDied","Data":"1195abf428bcf93e894c57d7492ba6e107928ad972577edf19b9c7b635f36d37"}
Sep 08 16:55:08.454119 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.454070    2557 generic.go:356] "Generic (PLEG): container finished" podID="5b87e294-ec93-4f11-867b-04e176f28a03" containerID="1195abf428bcf93e894c57d7492ba6e107928ad972577edf19b9c7b635f36d37" exitCode=0
Sep 08 16:55:08.455796 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.455771    2557 generic.go:356] "Generic (PLEG): container finished" podID="f839060a-9e2a-4042-adaa-04915065489f" containerID="b14f5344d69e17cae6a52058e85fe477a736fd01f76baaab754514549ac45fe0" exitCode=0
Sep 08 16:55:08.455941 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.455807    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd" event={"ID":"f839060a-9e2a-4042-adaa-04915065489f","Type":"ContainerDied","Data":"b14f5344d69e17cae6a52058e85fe477a736fd01f76baaab754514549ac45fe0"}
Sep 08 16:55:08.455941 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.455860    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd" event={"ID":"f839060a-9e2a-4042-adaa-04915065489f","Type":"ContainerDied","Data":"ecf773c6dac79c8ed6c8dc59dc4504deca41b6d7100fd202fb1b5dfe626d5441"}
Sep 08 16:55:08.455941 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.455884    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecf773c6dac79c8ed6c8dc59dc4504deca41b6d7100fd202fb1b5dfe626d5441"
Sep 08 16:55:08.456341 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.456214    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7818c2-9bm74"]
Sep 08 16:55:08.466356 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.466330    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd"
Sep 08 16:55:08.470147 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.470117    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n"
Sep 08 16:55:08.544385 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.544348    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b87e294-ec93-4f11-867b-04e176f28a03-plugins-config-volume\") pod \"5b87e294-ec93-4f11-867b-04e176f28a03\" (UID: \"5b87e294-ec93-4f11-867b-04e176f28a03\") "
Sep 08 16:55:08.544568 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.544423    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsp4r\" (UniqueName: \"kubernetes.io/projected/f839060a-9e2a-4042-adaa-04915065489f-kube-api-access-lsp4r\") pod \"f839060a-9e2a-4042-adaa-04915065489f\" (UID: \"f839060a-9e2a-4042-adaa-04915065489f\") "
Sep 08 16:55:08.544568 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.544447    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f839060a-9e2a-4042-adaa-04915065489f-plugins-config-volume\") pod \"f839060a-9e2a-4042-adaa-04915065489f\" (UID: \"f839060a-9e2a-4042-adaa-04915065489f\") "
Sep 08 16:55:08.544568 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.544467    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwlc8\" (UniqueName: \"kubernetes.io/projected/5b87e294-ec93-4f11-867b-04e176f28a03-kube-api-access-mwlc8\") pod \"5b87e294-ec93-4f11-867b-04e176f28a03\" (UID: \"5b87e294-ec93-4f11-867b-04e176f28a03\") "
Sep 08 16:55:08.544795 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.544770    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b87e294-ec93-4f11-867b-04e176f28a03-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5b87e294-ec93-4f11-867b-04e176f28a03" (UID: "5b87e294-ec93-4f11-867b-04e176f28a03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:55:08.544883 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.544825    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f839060a-9e2a-4042-adaa-04915065489f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f839060a-9e2a-4042-adaa-04915065489f" (UID: "f839060a-9e2a-4042-adaa-04915065489f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:55:08.546864 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.546814    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b87e294-ec93-4f11-867b-04e176f28a03-kube-api-access-mwlc8" (OuterVolumeSpecName: "kube-api-access-mwlc8") pod "5b87e294-ec93-4f11-867b-04e176f28a03" (UID: "5b87e294-ec93-4f11-867b-04e176f28a03"). InnerVolumeSpecName "kube-api-access-mwlc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:55:08.546864 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.546846    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f839060a-9e2a-4042-adaa-04915065489f-kube-api-access-lsp4r" (OuterVolumeSpecName: "kube-api-access-lsp4r") pod "f839060a-9e2a-4042-adaa-04915065489f" (UID: "f839060a-9e2a-4042-adaa-04915065489f"). InnerVolumeSpecName "kube-api-access-lsp4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:55:08.645424 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.645381    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b87e294-ec93-4f11-867b-04e176f28a03-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:08.645424 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.645418    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsp4r\" (UniqueName: \"kubernetes.io/projected/f839060a-9e2a-4042-adaa-04915065489f-kube-api-access-lsp4r\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:08.645424 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.645428    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f839060a-9e2a-4042-adaa-04915065489f-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:08.645424 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:08.645436    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwlc8\" (UniqueName: \"kubernetes.io/projected/5b87e294-ec93-4f11-867b-04e176f28a03-kube-api-access-mwlc8\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:09.348031 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:09.347985    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:55:09.461977 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:09.461944    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n"
Sep 08 16:55:09.462180 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:09.461980    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n" event={"ID":"5b87e294-ec93-4f11-867b-04e176f28a03","Type":"ContainerDied","Data":"b66759344a96258a04b20c95a54be25050158ab4f6941f4e50525e6a1cabfb9c"}
Sep 08 16:55:09.462180 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:09.462037    2557 scope.go:117] "RemoveContainer" containerID="1195abf428bcf93e894c57d7492ba6e107928ad972577edf19b9c7b635f36d37"
Sep 08 16:55:09.462301 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:09.462245    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd"
Sep 08 16:55:09.488873 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:09.488816    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd"]
Sep 08 16:55:09.493855 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:09.493806    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7818c2-86b68c8495-kg5nd"]
Sep 08 16:55:09.512728 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:09.512688    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n"]
Sep 08 16:55:09.515550 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:09.515519    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7818c2-69559c6bf4-6hn7n"]
Sep 08 16:55:10.353952 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:10.353916    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b87e294-ec93-4f11-867b-04e176f28a03" path="/var/lib/kubelet/pods/5b87e294-ec93-4f11-867b-04e176f28a03/volumes"
Sep 08 16:55:10.354335 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:10.354246    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e204bded-d118-4781-9275-bf23a80d817b" path="/var/lib/kubelet/pods/e204bded-d118-4781-9275-bf23a80d817b/volumes"
Sep 08 16:55:10.354551 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:10.354537    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f839060a-9e2a-4042-adaa-04915065489f" path="/var/lib/kubelet/pods/f839060a-9e2a-4042-adaa-04915065489f/volumes"
Sep 08 16:55:14.026334 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:14.026294    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"]
Sep 08 16:55:15.072558 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:15.072509    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"]
Sep 08 16:55:16.577288 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:16.577248    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1df9f8-8lkbj"]
Sep 08 16:55:16.577676 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:16.577660    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f839060a-9e2a-4042-adaa-04915065489f" containerName="payload-processing"
Sep 08 16:55:16.577736 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:16.577678    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f839060a-9e2a-4042-adaa-04915065489f" containerName="payload-processing"
Sep 08 16:55:16.577736 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:16.577689    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b87e294-ec93-4f11-867b-04e176f28a03" containerName="payload-pre-processing"
Sep 08 16:55:16.577736 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:16.577695    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b87e294-ec93-4f11-867b-04e176f28a03" containerName="payload-pre-processing"
Sep 08 16:55:16.577736 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:16.577705    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e204bded-d118-4781-9275-bf23a80d817b" containerName="revoke-keys"
Sep 08 16:55:16.577736 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:16.577710    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e204bded-d118-4781-9275-bf23a80d817b" containerName="revoke-keys"
Sep 08 16:55:16.577935 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:16.577774    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f839060a-9e2a-4042-adaa-04915065489f" containerName="payload-processing"
Sep 08 16:55:16.577935 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:16.577786    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e204bded-d118-4781-9275-bf23a80d817b" containerName="revoke-keys"
Sep 08 16:55:16.577935 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:16.577793    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b87e294-ec93-4f11-867b-04e176f28a03" containerName="payload-pre-processing"
Sep 08 16:55:16.580951 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:16.580935    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1df9f8-8lkbj"
Sep 08 16:55:16.584988 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:16.584963    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-886jf\""
Sep 08 16:55:16.585397 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:16.585382    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:55:16.607003 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:16.606969    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1df9f8-8lkbj"]
Sep 08 16:55:16.648254 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:16.648229    2557 scope.go:117] "RemoveContainer" containerID="06167e2c3d0a044c20d017a9de0a08d347a350032edc9c9223a50de44d502030"
Sep 08 16:55:16.715727 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:16.715685    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ff8e34-da6d-4228-bf27-fa9e5b83fefc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1df9f8-8lkbj\" (UID: \"e2ff8e34-da6d-4228-bf27-fa9e5b83fefc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1df9f8-8lkbj"
Sep 08 16:55:16.816292 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:16.816250    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ff8e34-da6d-4228-bf27-fa9e5b83fefc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1df9f8-8lkbj\" (UID: \"e2ff8e34-da6d-4228-bf27-fa9e5b83fefc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1df9f8-8lkbj"
Sep 08 16:55:16.816987 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:16.816961    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ff8e34-da6d-4228-bf27-fa9e5b83fefc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1df9f8-8lkbj\" (UID: \"e2ff8e34-da6d-4228-bf27-fa9e5b83fefc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1df9f8-8lkbj"
Sep 08 16:55:16.891454 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:16.891356    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1df9f8-8lkbj"
Sep 08 16:55:17.174369 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:17.174340    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1df9f8-8lkbj"]
Sep 08 16:55:17.177254 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:55:17.177222    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ff8e34_da6d_4228_bf27_fa9e5b83fefc.slice/crio-f0e5633e3e88721d80464f470932bcf8bd232c98a551489cdbf7c6b0e7c531ed WatchSource:0}: Error finding container f0e5633e3e88721d80464f470932bcf8bd232c98a551489cdbf7c6b0e7c531ed: Status 404 returned error can't find the container with id f0e5633e3e88721d80464f470932bcf8bd232c98a551489cdbf7c6b0e7c531ed
Sep 08 16:55:17.507676 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:17.507587    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1df9f8-8lkbj" event={"ID":"e2ff8e34-da6d-4228-bf27-fa9e5b83fefc","Type":"ContainerStarted","Data":"beef2f6802fa11ac3ba11e406b919a3a5d33bc833080cb35dfdbdf89301d0b50"}
Sep 08 16:55:17.507676 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:17.507637    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1df9f8-8lkbj" event={"ID":"e2ff8e34-da6d-4228-bf27-fa9e5b83fefc","Type":"ContainerStarted","Data":"f0e5633e3e88721d80464f470932bcf8bd232c98a551489cdbf7c6b0e7c531ed"}
Sep 08 16:55:18.514239 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:18.514205    2557 generic.go:356] "Generic (PLEG): container finished" podID="e2ff8e34-da6d-4228-bf27-fa9e5b83fefc" containerID="beef2f6802fa11ac3ba11e406b919a3a5d33bc833080cb35dfdbdf89301d0b50" exitCode=0
Sep 08 16:55:18.514705 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:18.514250    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1df9f8-8lkbj" event={"ID":"e2ff8e34-da6d-4228-bf27-fa9e5b83fefc","Type":"ContainerDied","Data":"beef2f6802fa11ac3ba11e406b919a3a5d33bc833080cb35dfdbdf89301d0b50"}
Sep 08 16:55:19.655848 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:19.655806    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1df9f8-8lkbj"
Sep 08 16:55:19.843600 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:19.843511    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ff8e34-da6d-4228-bf27-fa9e5b83fefc-maas-api-service-ca\") pod \"e2ff8e34-da6d-4228-bf27-fa9e5b83fefc\" (UID: \"e2ff8e34-da6d-4228-bf27-fa9e5b83fefc\") "
Sep 08 16:55:19.843890 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:19.843868    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2ff8e34-da6d-4228-bf27-fa9e5b83fefc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e2ff8e34-da6d-4228-bf27-fa9e5b83fefc" (UID: "e2ff8e34-da6d-4228-bf27-fa9e5b83fefc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:55:19.945076 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:19.945032    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ff8e34-da6d-4228-bf27-fa9e5b83fefc-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:20.387676 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:55:20.387637    2557 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ff8e34_da6d_4228_bf27_fa9e5b83fefc.slice\": RecentStats: unable to find data in memory cache]"
Sep 08 16:55:20.526181 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:20.526148    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1df9f8-8lkbj"
Sep 08 16:55:20.526352 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:20.526149    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1df9f8-8lkbj" event={"ID":"e2ff8e34-da6d-4228-bf27-fa9e5b83fefc","Type":"ContainerDied","Data":"f0e5633e3e88721d80464f470932bcf8bd232c98a551489cdbf7c6b0e7c531ed"}
Sep 08 16:55:20.526352 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:20.526296    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0e5633e3e88721d80464f470932bcf8bd232c98a551489cdbf7c6b0e7c531ed"
Sep 08 16:55:21.571548 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:21.571513    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf"]
Sep 08 16:55:21.572056 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:21.571969    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ff8e34-da6d-4228-bf27-fa9e5b83fefc" containerName="revoke-keys"
Sep 08 16:55:21.572056 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:21.571981    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ff8e34-da6d-4228-bf27-fa9e5b83fefc" containerName="revoke-keys"
Sep 08 16:55:21.572175 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:21.572059    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2ff8e34-da6d-4228-bf27-fa9e5b83fefc" containerName="revoke-keys"
Sep 08 16:55:21.574927 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:21.574906    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf"
Sep 08 16:55:21.577581 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:21.577551    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d2b1cc-dockercfg-994pq\""
Sep 08 16:55:21.577722 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:21.577685    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d2b1cc\""
Sep 08 16:55:21.610766 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:21.610732    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf"]
Sep 08 16:55:21.668334 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:21.668289    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmmg9\" (UniqueName: \"kubernetes.io/projected/c2189b15-dc78-4597-a0ff-1bec9bf395ad-kube-api-access-bmmg9\") pod \"payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf\" (UID: \"c2189b15-dc78-4597-a0ff-1bec9bf395ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf"
Sep 08 16:55:21.668530 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:21.668404    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2189b15-dc78-4597-a0ff-1bec9bf395ad-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf\" (UID: \"c2189b15-dc78-4597-a0ff-1bec9bf395ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf"
Sep 08 16:55:21.769707 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:21.769664    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2189b15-dc78-4597-a0ff-1bec9bf395ad-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf\" (UID: \"c2189b15-dc78-4597-a0ff-1bec9bf395ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf"
Sep 08 16:55:21.769951 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:21.769768    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmmg9\" (UniqueName: \"kubernetes.io/projected/c2189b15-dc78-4597-a0ff-1bec9bf395ad-kube-api-access-bmmg9\") pod \"payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf\" (UID: \"c2189b15-dc78-4597-a0ff-1bec9bf395ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf"
Sep 08 16:55:21.770540 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:21.770513    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2189b15-dc78-4597-a0ff-1bec9bf395ad-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf\" (UID: \"c2189b15-dc78-4597-a0ff-1bec9bf395ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf"
Sep 08 16:55:21.784768 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:21.784732    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmmg9\" (UniqueName: \"kubernetes.io/projected/c2189b15-dc78-4597-a0ff-1bec9bf395ad-kube-api-access-bmmg9\") pod \"payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf\" (UID: \"c2189b15-dc78-4597-a0ff-1bec9bf395ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf"
Sep 08 16:55:21.887758 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:21.887663    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf"
Sep 08 16:55:22.168027 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:22.168001    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf"]
Sep 08 16:55:22.170086 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:55:22.170055    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2189b15_dc78_4597_a0ff_1bec9bf395ad.slice/crio-339398cc7c2cddafcbcab9e2868fffb38d7665337b35856a98a0bf4557bda853 WatchSource:0}: Error finding container 339398cc7c2cddafcbcab9e2868fffb38d7665337b35856a98a0bf4557bda853: Status 404 returned error can't find the container with id 339398cc7c2cddafcbcab9e2868fffb38d7665337b35856a98a0bf4557bda853
Sep 08 16:55:22.539764 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:22.539656    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf" event={"ID":"c2189b15-dc78-4597-a0ff-1bec9bf395ad","Type":"ContainerStarted","Data":"736461633af7ad8fb707f25139b3c74c27b096236352798081996584148a52ef"}
Sep 08 16:55:22.539764 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:22.539713    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf" event={"ID":"c2189b15-dc78-4597-a0ff-1bec9bf395ad","Type":"ContainerStarted","Data":"339398cc7c2cddafcbcab9e2868fffb38d7665337b35856a98a0bf4557bda853"}
Sep 08 16:55:22.540002 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:22.539824    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf"
Sep 08 16:55:22.585564 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:22.585483    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf" podStartSLOduration=1.5854660200000001 podStartE2EDuration="1.58546602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:55:22.584904273 +0000 UTC m=+1506.747875721" watchObservedRunningTime="2026-09-08 16:55:22.58546602 +0000 UTC m=+1506.748437472"
Sep 08 16:55:25.874671 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:25.874631    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:55:27.688795 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:27.688755    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d"]
Sep 08 16:55:27.689377 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:27.689042    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d" podUID="24250dbd-430b-46ea-bf70-0baa9072373e" containerName="payload-processing" containerID="cri-o://9fc2283be5f7477b7a1ab12a92bd05e5f0ee77ee6e53e5396e8771e7ba6097c9" gracePeriod=30
Sep 08 16:55:27.723574 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:27.723540    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd"]
Sep 08 16:55:27.723766 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:27.723744    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd" podUID="4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e" containerName="payload-pre-processing" containerID="cri-o://e0f77d02bf7ee553407d44bcdb8bb5bd7849347ae86404f331cee0cee7defe1f" gracePeriod=30
Sep 08 16:55:28.019092 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.019066    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d"
Sep 08 16:55:28.021872 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.021819    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd"
Sep 08 16:55:28.098651 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.098611    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1df9f8-8lkbj"]
Sep 08 16:55:28.106743 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.106711    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1df9f8-8lkbj"]
Sep 08 16:55:28.124182 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.124152    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqh8p\" (UniqueName: \"kubernetes.io/projected/4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e-kube-api-access-fqh8p\") pod \"4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e\" (UID: \"4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e\") "
Sep 08 16:55:28.124182 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.124191    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e-plugins-config-volume\") pod \"4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e\" (UID: \"4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e\") "
Sep 08 16:55:28.124455 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.124237    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwvcf\" (UniqueName: \"kubernetes.io/projected/24250dbd-430b-46ea-bf70-0baa9072373e-kube-api-access-cwvcf\") pod \"24250dbd-430b-46ea-bf70-0baa9072373e\" (UID: \"24250dbd-430b-46ea-bf70-0baa9072373e\") "
Sep 08 16:55:28.124455 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.124322    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24250dbd-430b-46ea-bf70-0baa9072373e-plugins-config-volume\") pod \"24250dbd-430b-46ea-bf70-0baa9072373e\" (UID: \"24250dbd-430b-46ea-bf70-0baa9072373e\") "
Sep 08 16:55:28.124707 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.124679    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e" (UID: "4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:55:28.124775 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.124760    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24250dbd-430b-46ea-bf70-0baa9072373e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "24250dbd-430b-46ea-bf70-0baa9072373e" (UID: "24250dbd-430b-46ea-bf70-0baa9072373e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:55:28.126615 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.126591    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e-kube-api-access-fqh8p" (OuterVolumeSpecName: "kube-api-access-fqh8p") pod "4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e" (UID: "4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e"). InnerVolumeSpecName "kube-api-access-fqh8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:55:28.126705 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.126605    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24250dbd-430b-46ea-bf70-0baa9072373e-kube-api-access-cwvcf" (OuterVolumeSpecName: "kube-api-access-cwvcf") pod "24250dbd-430b-46ea-bf70-0baa9072373e" (UID: "24250dbd-430b-46ea-bf70-0baa9072373e"). InnerVolumeSpecName "kube-api-access-cwvcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:55:28.225647 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.225606    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqh8p\" (UniqueName: \"kubernetes.io/projected/4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e-kube-api-access-fqh8p\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:28.225647 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.225637    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:28.225647 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.225646    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwvcf\" (UniqueName: \"kubernetes.io/projected/24250dbd-430b-46ea-bf70-0baa9072373e-kube-api-access-cwvcf\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:28.225647 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.225655    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24250dbd-430b-46ea-bf70-0baa9072373e-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:28.346254 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.346168    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2ff8e34-da6d-4228-bf27-fa9e5b83fefc" path="/var/lib/kubelet/pods/e2ff8e34-da6d-4228-bf27-fa9e5b83fefc/volumes"
Sep 08 16:55:28.571565 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.571524    2557 generic.go:356] "Generic (PLEG): container finished" podID="24250dbd-430b-46ea-bf70-0baa9072373e" containerID="9fc2283be5f7477b7a1ab12a92bd05e5f0ee77ee6e53e5396e8771e7ba6097c9" exitCode=0
Sep 08 16:55:28.571565 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.571570    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d"
Sep 08 16:55:28.571865 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.571611    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d" event={"ID":"24250dbd-430b-46ea-bf70-0baa9072373e","Type":"ContainerDied","Data":"9fc2283be5f7477b7a1ab12a92bd05e5f0ee77ee6e53e5396e8771e7ba6097c9"}
Sep 08 16:55:28.571865 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.571674    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d" event={"ID":"24250dbd-430b-46ea-bf70-0baa9072373e","Type":"ContainerDied","Data":"cfe9530492f42e149ff7457a52d85c1f3940234b726584d33b96d054cd3ef943"}
Sep 08 16:55:28.571865 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.571704    2557 scope.go:117] "RemoveContainer" containerID="9fc2283be5f7477b7a1ab12a92bd05e5f0ee77ee6e53e5396e8771e7ba6097c9"
Sep 08 16:55:28.573907 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.573877    2557 generic.go:356] "Generic (PLEG): container finished" podID="4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e" containerID="e0f77d02bf7ee553407d44bcdb8bb5bd7849347ae86404f331cee0cee7defe1f" exitCode=0
Sep 08 16:55:28.574037 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.573949    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd" event={"ID":"4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e","Type":"ContainerDied","Data":"e0f77d02bf7ee553407d44bcdb8bb5bd7849347ae86404f331cee0cee7defe1f"}
Sep 08 16:55:28.574037 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.573999    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd" event={"ID":"4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e","Type":"ContainerDied","Data":"e64f9fe04569c34b49ad06f31f3a77246054f1577271264ce2c678ce57601400"}
Sep 08 16:55:28.574159 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.574071    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd"
Sep 08 16:55:28.581947 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.581925    2557 scope.go:117] "RemoveContainer" containerID="9fc2283be5f7477b7a1ab12a92bd05e5f0ee77ee6e53e5396e8771e7ba6097c9"
Sep 08 16:55:28.582300 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:55:28.582280    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fc2283be5f7477b7a1ab12a92bd05e5f0ee77ee6e53e5396e8771e7ba6097c9\": container with ID starting with 9fc2283be5f7477b7a1ab12a92bd05e5f0ee77ee6e53e5396e8771e7ba6097c9 not found: ID does not exist" containerID="9fc2283be5f7477b7a1ab12a92bd05e5f0ee77ee6e53e5396e8771e7ba6097c9"
Sep 08 16:55:28.582393 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.582314    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fc2283be5f7477b7a1ab12a92bd05e5f0ee77ee6e53e5396e8771e7ba6097c9"} err="failed to get container status \"9fc2283be5f7477b7a1ab12a92bd05e5f0ee77ee6e53e5396e8771e7ba6097c9\": rpc error: code = NotFound desc = could not find container \"9fc2283be5f7477b7a1ab12a92bd05e5f0ee77ee6e53e5396e8771e7ba6097c9\": container with ID starting with 9fc2283be5f7477b7a1ab12a92bd05e5f0ee77ee6e53e5396e8771e7ba6097c9 not found: ID does not exist"
Sep 08 16:55:28.582393 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.582338    2557 scope.go:117] "RemoveContainer" containerID="e0f77d02bf7ee553407d44bcdb8bb5bd7849347ae86404f331cee0cee7defe1f"
Sep 08 16:55:28.591239 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.591209    2557 scope.go:117] "RemoveContainer" containerID="e0f77d02bf7ee553407d44bcdb8bb5bd7849347ae86404f331cee0cee7defe1f"
Sep 08 16:55:28.591627 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:55:28.591604    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0f77d02bf7ee553407d44bcdb8bb5bd7849347ae86404f331cee0cee7defe1f\": container with ID starting with e0f77d02bf7ee553407d44bcdb8bb5bd7849347ae86404f331cee0cee7defe1f not found: ID does not exist" containerID="e0f77d02bf7ee553407d44bcdb8bb5bd7849347ae86404f331cee0cee7defe1f"
Sep 08 16:55:28.591722 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.591634    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0f77d02bf7ee553407d44bcdb8bb5bd7849347ae86404f331cee0cee7defe1f"} err="failed to get container status \"e0f77d02bf7ee553407d44bcdb8bb5bd7849347ae86404f331cee0cee7defe1f\": rpc error: code = NotFound desc = could not find container \"e0f77d02bf7ee553407d44bcdb8bb5bd7849347ae86404f331cee0cee7defe1f\": container with ID starting with e0f77d02bf7ee553407d44bcdb8bb5bd7849347ae86404f331cee0cee7defe1f not found: ID does not exist"
Sep 08 16:55:28.610338 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.610252    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d"]
Sep 08 16:55:28.618958 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.618932    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1df9f8-b78dd8749-n872d"]
Sep 08 16:55:28.655335 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.655296    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd"]
Sep 08 16:55:28.666395 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:28.666364    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1df9f8-7486985746-nqtjd"]
Sep 08 16:55:29.046966 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.046928    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w" podUID="40703dcd-7963-415f-862c-922f47f8edbc" containerName="main" containerID="cri-o://713e7f0017b75ee16abc8c2e32c36054214999cdc3fac27304fb4461849bcf5d" gracePeriod=45
Sep 08 16:55:29.302663 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.302593    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:55:29.335005 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.334974    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40703dcd-7963-415f-862c-922f47f8edbc-tmp-dir\") pod \"40703dcd-7963-415f-862c-922f47f8edbc\" (UID: \"40703dcd-7963-415f-862c-922f47f8edbc\") "
Sep 08 16:55:29.335191 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.335029    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40703dcd-7963-415f-862c-922f47f8edbc-dshm\") pod \"40703dcd-7963-415f-862c-922f47f8edbc\" (UID: \"40703dcd-7963-415f-862c-922f47f8edbc\") "
Sep 08 16:55:29.335191 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.335050    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40703dcd-7963-415f-862c-922f47f8edbc-model-cache\") pod \"40703dcd-7963-415f-862c-922f47f8edbc\" (UID: \"40703dcd-7963-415f-862c-922f47f8edbc\") "
Sep 08 16:55:29.335191 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.335070    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s98br\" (UniqueName: \"kubernetes.io/projected/40703dcd-7963-415f-862c-922f47f8edbc-kube-api-access-s98br\") pod \"40703dcd-7963-415f-862c-922f47f8edbc\" (UID: \"40703dcd-7963-415f-862c-922f47f8edbc\") "
Sep 08 16:55:29.335191 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.335095    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40703dcd-7963-415f-862c-922f47f8edbc-tls-certs\") pod \"40703dcd-7963-415f-862c-922f47f8edbc\" (UID: \"40703dcd-7963-415f-862c-922f47f8edbc\") "
Sep 08 16:55:29.335191 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.335168    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40703dcd-7963-415f-862c-922f47f8edbc-home\") pod \"40703dcd-7963-415f-862c-922f47f8edbc\" (UID: \"40703dcd-7963-415f-862c-922f47f8edbc\") "
Sep 08 16:55:29.335522 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.335286    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40703dcd-7963-415f-862c-922f47f8edbc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "40703dcd-7963-415f-862c-922f47f8edbc" (UID: "40703dcd-7963-415f-862c-922f47f8edbc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:55:29.335522 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.335302    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40703dcd-7963-415f-862c-922f47f8edbc-model-cache" (OuterVolumeSpecName: "model-cache") pod "40703dcd-7963-415f-862c-922f47f8edbc" (UID: "40703dcd-7963-415f-862c-922f47f8edbc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:55:29.335522 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.335387    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40703dcd-7963-415f-862c-922f47f8edbc-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:29.335522 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.335405    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40703dcd-7963-415f-862c-922f47f8edbc-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:29.335654 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.335522    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40703dcd-7963-415f-862c-922f47f8edbc-home" (OuterVolumeSpecName: "home") pod "40703dcd-7963-415f-862c-922f47f8edbc" (UID: "40703dcd-7963-415f-862c-922f47f8edbc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:55:29.337366 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.337345    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40703dcd-7963-415f-862c-922f47f8edbc-dshm" (OuterVolumeSpecName: "dshm") pod "40703dcd-7963-415f-862c-922f47f8edbc" (UID: "40703dcd-7963-415f-862c-922f47f8edbc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:55:29.337468 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.337365    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40703dcd-7963-415f-862c-922f47f8edbc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40703dcd-7963-415f-862c-922f47f8edbc" (UID: "40703dcd-7963-415f-862c-922f47f8edbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:55:29.337468 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.337400    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40703dcd-7963-415f-862c-922f47f8edbc-kube-api-access-s98br" (OuterVolumeSpecName: "kube-api-access-s98br") pod "40703dcd-7963-415f-862c-922f47f8edbc" (UID: "40703dcd-7963-415f-862c-922f47f8edbc"). InnerVolumeSpecName "kube-api-access-s98br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:55:29.436071 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.436036    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40703dcd-7963-415f-862c-922f47f8edbc-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:29.436071 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.436066    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s98br\" (UniqueName: \"kubernetes.io/projected/40703dcd-7963-415f-862c-922f47f8edbc-kube-api-access-s98br\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:29.436071 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.436078    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40703dcd-7963-415f-862c-922f47f8edbc-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:29.436313 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.436087    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40703dcd-7963-415f-862c-922f47f8edbc-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:29.581998 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.581892    2557 generic.go:356] "Generic (PLEG): container finished" podID="40703dcd-7963-415f-862c-922f47f8edbc" containerID="713e7f0017b75ee16abc8c2e32c36054214999cdc3fac27304fb4461849bcf5d" exitCode=0
Sep 08 16:55:29.581998 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.581934    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"
Sep 08 16:55:29.581998 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.581984    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w" event={"ID":"40703dcd-7963-415f-862c-922f47f8edbc","Type":"ContainerDied","Data":"713e7f0017b75ee16abc8c2e32c36054214999cdc3fac27304fb4461849bcf5d"}
Sep 08 16:55:29.582352 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.582035    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w" event={"ID":"40703dcd-7963-415f-862c-922f47f8edbc","Type":"ContainerDied","Data":"32cd136c55b2e7fcb7e57f2c9468093064714258ebbf15accdbce15dabd65fac"}
Sep 08 16:55:29.582352 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.582060    2557 scope.go:117] "RemoveContainer" containerID="713e7f0017b75ee16abc8c2e32c36054214999cdc3fac27304fb4461849bcf5d"
Sep 08 16:55:29.591785 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.591767    2557 scope.go:117] "RemoveContainer" containerID="713e7f0017b75ee16abc8c2e32c36054214999cdc3fac27304fb4461849bcf5d"
Sep 08 16:55:29.592128 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:55:29.592105    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"713e7f0017b75ee16abc8c2e32c36054214999cdc3fac27304fb4461849bcf5d\": container with ID starting with 713e7f0017b75ee16abc8c2e32c36054214999cdc3fac27304fb4461849bcf5d not found: ID does not exist" containerID="713e7f0017b75ee16abc8c2e32c36054214999cdc3fac27304fb4461849bcf5d"
Sep 08 16:55:29.592190 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.592136    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713e7f0017b75ee16abc8c2e32c36054214999cdc3fac27304fb4461849bcf5d"} err="failed to get container status \"713e7f0017b75ee16abc8c2e32c36054214999cdc3fac27304fb4461849bcf5d\": rpc error: code = NotFound desc = could not find container \"713e7f0017b75ee16abc8c2e32c36054214999cdc3fac27304fb4461849bcf5d\": container with ID starting with 713e7f0017b75ee16abc8c2e32c36054214999cdc3fac27304fb4461849bcf5d not found: ID does not exist"
Sep 08 16:55:29.612304 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.612269    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"]
Sep 08 16:55:29.626340 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:29.626304    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-608268/rate-test-model-608268-kserve-6cd9886955-f7t4w"]
Sep 08 16:55:30.093616 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.093576    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd" podUID="ac553768-4a41-43c5-aef7-0e9ae8cb726d" containerName="main" containerID="cri-o://77e83dd3919e8009fc50260bad0f2bc8c59a6a061321df05e4ff71e8dfd561c4" gracePeriod=45
Sep 08 16:55:30.345819 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.345755    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24250dbd-430b-46ea-bf70-0baa9072373e" path="/var/lib/kubelet/pods/24250dbd-430b-46ea-bf70-0baa9072373e/volumes"
Sep 08 16:55:30.346177 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.346162    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40703dcd-7963-415f-862c-922f47f8edbc" path="/var/lib/kubelet/pods/40703dcd-7963-415f-862c-922f47f8edbc/volumes"
Sep 08 16:55:30.346479 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.346467    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e" path="/var/lib/kubelet/pods/4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e/volumes"
Sep 08 16:55:30.365193 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.365161    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:55:30.444064 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.444030    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac553768-4a41-43c5-aef7-0e9ae8cb726d-home\") pod \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\" (UID: \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\") "
Sep 08 16:55:30.444064 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.444064    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac553768-4a41-43c5-aef7-0e9ae8cb726d-tmp-dir\") pod \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\" (UID: \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\") "
Sep 08 16:55:30.444297 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.444093    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4knkw\" (UniqueName: \"kubernetes.io/projected/ac553768-4a41-43c5-aef7-0e9ae8cb726d-kube-api-access-4knkw\") pod \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\" (UID: \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\") "
Sep 08 16:55:30.444297 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.444145    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac553768-4a41-43c5-aef7-0e9ae8cb726d-model-cache\") pod \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\" (UID: \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\") "
Sep 08 16:55:30.444297 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.444172    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac553768-4a41-43c5-aef7-0e9ae8cb726d-dshm\") pod \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\" (UID: \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\") "
Sep 08 16:55:30.444434 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.444293    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac553768-4a41-43c5-aef7-0e9ae8cb726d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ac553768-4a41-43c5-aef7-0e9ae8cb726d" (UID: "ac553768-4a41-43c5-aef7-0e9ae8cb726d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:55:30.444434 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.444347    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac553768-4a41-43c5-aef7-0e9ae8cb726d-home" (OuterVolumeSpecName: "home") pod "ac553768-4a41-43c5-aef7-0e9ae8cb726d" (UID: "ac553768-4a41-43c5-aef7-0e9ae8cb726d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:55:30.444434 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.444358    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac553768-4a41-43c5-aef7-0e9ae8cb726d-tls-certs\") pod \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\" (UID: \"ac553768-4a41-43c5-aef7-0e9ae8cb726d\") "
Sep 08 16:55:30.444533 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.444440    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac553768-4a41-43c5-aef7-0e9ae8cb726d-model-cache" (OuterVolumeSpecName: "model-cache") pod "ac553768-4a41-43c5-aef7-0e9ae8cb726d" (UID: "ac553768-4a41-43c5-aef7-0e9ae8cb726d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:55:30.444650 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.444630    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac553768-4a41-43c5-aef7-0e9ae8cb726d-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:30.444693 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.444681    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac553768-4a41-43c5-aef7-0e9ae8cb726d-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:30.444752 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.444699    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac553768-4a41-43c5-aef7-0e9ae8cb726d-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:30.446548 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.446524    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac553768-4a41-43c5-aef7-0e9ae8cb726d-dshm" (OuterVolumeSpecName: "dshm") pod "ac553768-4a41-43c5-aef7-0e9ae8cb726d" (UID: "ac553768-4a41-43c5-aef7-0e9ae8cb726d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:55:30.446644 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.446597    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac553768-4a41-43c5-aef7-0e9ae8cb726d-kube-api-access-4knkw" (OuterVolumeSpecName: "kube-api-access-4knkw") pod "ac553768-4a41-43c5-aef7-0e9ae8cb726d" (UID: "ac553768-4a41-43c5-aef7-0e9ae8cb726d"). InnerVolumeSpecName "kube-api-access-4knkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:55:30.446644 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.446609    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac553768-4a41-43c5-aef7-0e9ae8cb726d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ac553768-4a41-43c5-aef7-0e9ae8cb726d" (UID: "ac553768-4a41-43c5-aef7-0e9ae8cb726d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:55:30.545314 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.545279    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac553768-4a41-43c5-aef7-0e9ae8cb726d-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:30.545314 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.545312    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4knkw\" (UniqueName: \"kubernetes.io/projected/ac553768-4a41-43c5-aef7-0e9ae8cb726d-kube-api-access-4knkw\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:30.545314 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.545321    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac553768-4a41-43c5-aef7-0e9ae8cb726d-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:30.591350 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.591314    2557 generic.go:356] "Generic (PLEG): container finished" podID="ac553768-4a41-43c5-aef7-0e9ae8cb726d" containerID="77e83dd3919e8009fc50260bad0f2bc8c59a6a061321df05e4ff71e8dfd561c4" exitCode=0
Sep 08 16:55:30.591538 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.591386    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"
Sep 08 16:55:30.591538 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.591410    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd" event={"ID":"ac553768-4a41-43c5-aef7-0e9ae8cb726d","Type":"ContainerDied","Data":"77e83dd3919e8009fc50260bad0f2bc8c59a6a061321df05e4ff71e8dfd561c4"}
Sep 08 16:55:30.591538 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.591454    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd" event={"ID":"ac553768-4a41-43c5-aef7-0e9ae8cb726d","Type":"ContainerDied","Data":"880cbf3ce944d69c78ea7faceca232c6cf409e247faf2704ee2bd21a7b40e473"}
Sep 08 16:55:30.591538 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.591473    2557 scope.go:117] "RemoveContainer" containerID="77e83dd3919e8009fc50260bad0f2bc8c59a6a061321df05e4ff71e8dfd561c4"
Sep 08 16:55:30.601036 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.601016    2557 scope.go:117] "RemoveContainer" containerID="77e83dd3919e8009fc50260bad0f2bc8c59a6a061321df05e4ff71e8dfd561c4"
Sep 08 16:55:30.601374 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:55:30.601356    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77e83dd3919e8009fc50260bad0f2bc8c59a6a061321df05e4ff71e8dfd561c4\": container with ID starting with 77e83dd3919e8009fc50260bad0f2bc8c59a6a061321df05e4ff71e8dfd561c4 not found: ID does not exist" containerID="77e83dd3919e8009fc50260bad0f2bc8c59a6a061321df05e4ff71e8dfd561c4"
Sep 08 16:55:30.601445 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.601383    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77e83dd3919e8009fc50260bad0f2bc8c59a6a061321df05e4ff71e8dfd561c4"} err="failed to get container status \"77e83dd3919e8009fc50260bad0f2bc8c59a6a061321df05e4ff71e8dfd561c4\": rpc error: code = NotFound desc = could not find container \"77e83dd3919e8009fc50260bad0f2bc8c59a6a061321df05e4ff71e8dfd561c4\": container with ID starting with 77e83dd3919e8009fc50260bad0f2bc8c59a6a061321df05e4ff71e8dfd561c4 not found: ID does not exist"
Sep 08 16:55:30.621994 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.621961    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"]
Sep 08 16:55:30.625217 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:30.625191    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6b15ca/rate-test-model-6b15ca-kserve-bb79b8555-wqppd"]
Sep 08 16:55:32.345540 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:32.345491    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac553768-4a41-43c5-aef7-0e9ae8cb726d" path="/var/lib/kubelet/pods/ac553768-4a41-43c5-aef7-0e9ae8cb726d/volumes"
Sep 08 16:55:33.546933 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:33.546903    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf"
Sep 08 16:55:36.762325 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:36.762283    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:55:36.973594 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:36.973520    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e5401-wxnqm"]
Sep 08 16:55:36.974004 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:36.973978    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e" containerName="payload-pre-processing"
Sep 08 16:55:36.974004 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:36.974007    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e" containerName="payload-pre-processing"
Sep 08 16:55:36.974092 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:36.974018    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40703dcd-7963-415f-862c-922f47f8edbc" containerName="main"
Sep 08 16:55:36.974092 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:36.974024    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="40703dcd-7963-415f-862c-922f47f8edbc" containerName="main"
Sep 08 16:55:36.974092 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:36.974036    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24250dbd-430b-46ea-bf70-0baa9072373e" containerName="payload-processing"
Sep 08 16:55:36.974092 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:36.974042    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="24250dbd-430b-46ea-bf70-0baa9072373e" containerName="payload-processing"
Sep 08 16:55:36.974092 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:36.974051    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac553768-4a41-43c5-aef7-0e9ae8cb726d" containerName="main"
Sep 08 16:55:36.974092 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:36.974057    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac553768-4a41-43c5-aef7-0e9ae8cb726d" containerName="main"
Sep 08 16:55:36.974274 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:36.974110    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac553768-4a41-43c5-aef7-0e9ae8cb726d" containerName="main"
Sep 08 16:55:36.974274 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:36.974120    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="24250dbd-430b-46ea-bf70-0baa9072373e" containerName="payload-processing"
Sep 08 16:55:36.974274 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:36.974126    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ccf9b5f-ca47-49ee-a0a9-a6e58f15b17e" containerName="payload-pre-processing"
Sep 08 16:55:36.974274 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:36.974132    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="40703dcd-7963-415f-862c-922f47f8edbc" containerName="main"
Sep 08 16:55:36.978397 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:36.978375    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e5401-wxnqm"
Sep 08 16:55:36.981453 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:36.981428    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:55:36.981603 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:36.981462    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-886jf\""
Sep 08 16:55:36.993531 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:36.993505    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e5401-wxnqm"]
Sep 08 16:55:36.998689 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:36.998664    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57956374-67bc-48de-8d4d-f155ebef005e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6e5401-wxnqm\" (UID: \"57956374-67bc-48de-8d4d-f155ebef005e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e5401-wxnqm"
Sep 08 16:55:37.099757 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:37.099654    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57956374-67bc-48de-8d4d-f155ebef005e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6e5401-wxnqm\" (UID: \"57956374-67bc-48de-8d4d-f155ebef005e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e5401-wxnqm"
Sep 08 16:55:37.100282 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:37.100263    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57956374-67bc-48de-8d4d-f155ebef005e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6e5401-wxnqm\" (UID: \"57956374-67bc-48de-8d4d-f155ebef005e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e5401-wxnqm"
Sep 08 16:55:37.288420 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:37.288380    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e5401-wxnqm"
Sep 08 16:55:37.569369 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:37.569329    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e5401-wxnqm"]
Sep 08 16:55:37.573251 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:55:37.573218    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57956374_67bc_48de_8d4d_f155ebef005e.slice/crio-ed7204cf831deb0ece73729e74b710fbdf59f66351fa26edbb2a85a2c8eed086 WatchSource:0}: Error finding container ed7204cf831deb0ece73729e74b710fbdf59f66351fa26edbb2a85a2c8eed086: Status 404 returned error can't find the container with id ed7204cf831deb0ece73729e74b710fbdf59f66351fa26edbb2a85a2c8eed086
Sep 08 16:55:37.632507 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:37.632469    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e5401-wxnqm" event={"ID":"57956374-67bc-48de-8d4d-f155ebef005e","Type":"ContainerStarted","Data":"ed7204cf831deb0ece73729e74b710fbdf59f66351fa26edbb2a85a2c8eed086"}
Sep 08 16:55:38.638698 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:38.638595    2557 generic.go:356] "Generic (PLEG): container finished" podID="57956374-67bc-48de-8d4d-f155ebef005e" containerID="87a2342345641e3531fc573b39cf965c4ee77fe8e590c566f33cc3cf2cc25316" exitCode=0
Sep 08 16:55:38.639137 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:38.638696    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e5401-wxnqm" event={"ID":"57956374-67bc-48de-8d4d-f155ebef005e","Type":"ContainerDied","Data":"87a2342345641e3531fc573b39cf965c4ee77fe8e590c566f33cc3cf2cc25316"}
Sep 08 16:55:39.781612 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:39.781586    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e5401-wxnqm"
Sep 08 16:55:39.824590 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:39.824542    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57956374-67bc-48de-8d4d-f155ebef005e-maas-api-service-ca\") pod \"57956374-67bc-48de-8d4d-f155ebef005e\" (UID: \"57956374-67bc-48de-8d4d-f155ebef005e\") "
Sep 08 16:55:39.824918 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:39.824897    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57956374-67bc-48de-8d4d-f155ebef005e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "57956374-67bc-48de-8d4d-f155ebef005e" (UID: "57956374-67bc-48de-8d4d-f155ebef005e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:55:39.925341 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:39.925306    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57956374-67bc-48de-8d4d-f155ebef005e-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:40.652258 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:40.652147    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e5401-wxnqm" event={"ID":"57956374-67bc-48de-8d4d-f155ebef005e","Type":"ContainerDied","Data":"ed7204cf831deb0ece73729e74b710fbdf59f66351fa26edbb2a85a2c8eed086"}
Sep 08 16:55:40.652258 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:40.652256    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed7204cf831deb0ece73729e74b710fbdf59f66351fa26edbb2a85a2c8eed086"
Sep 08 16:55:40.652471 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:40.652341    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e5401-wxnqm"
Sep 08 16:55:48.421113 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.421080    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk"]
Sep 08 16:55:48.421513 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.421302    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk" podUID="84aaa926-730e-460d-bba8-36bfff099744" containerName="payload-processing" containerID="cri-o://125da4abeaca6239f6a66cbbaa260c1cafeccaee185b7b24c5764434f2586760" gracePeriod=30
Sep 08 16:55:48.444285 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.444257    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk"]
Sep 08 16:55:48.444491 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.444456    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk" podUID="71fadf7d-46e1-4a4a-abc9-8276185c31e6" containerName="payload-pre-processing" containerID="cri-o://4815347b806004b749d231e6078cc158b72af08ee93b20e2da8e643d2cd603a5" gracePeriod=30
Sep 08 16:55:48.698119 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.698001    2557 generic.go:356] "Generic (PLEG): container finished" podID="71fadf7d-46e1-4a4a-abc9-8276185c31e6" containerID="4815347b806004b749d231e6078cc158b72af08ee93b20e2da8e643d2cd603a5" exitCode=0
Sep 08 16:55:48.698119 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.698088    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk" event={"ID":"71fadf7d-46e1-4a4a-abc9-8276185c31e6","Type":"ContainerDied","Data":"4815347b806004b749d231e6078cc158b72af08ee93b20e2da8e643d2cd603a5"}
Sep 08 16:55:48.698325 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.698111    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk"
Sep 08 16:55:48.699770 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.699748    2557 generic.go:356] "Generic (PLEG): container finished" podID="84aaa926-730e-460d-bba8-36bfff099744" containerID="125da4abeaca6239f6a66cbbaa260c1cafeccaee185b7b24c5764434f2586760" exitCode=0
Sep 08 16:55:48.699894 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.699793    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk" event={"ID":"84aaa926-730e-460d-bba8-36bfff099744","Type":"ContainerDied","Data":"125da4abeaca6239f6a66cbbaa260c1cafeccaee185b7b24c5764434f2586760"}
Sep 08 16:55:48.699977 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.699907    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk" event={"ID":"84aaa926-730e-460d-bba8-36bfff099744","Type":"ContainerDied","Data":"805cae092cadb869d102b3c50c3db68632dd440a35217cd4d0cbd366c637f65d"}
Sep 08 16:55:48.699977 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.699930    2557 scope.go:117] "RemoveContainer" containerID="125da4abeaca6239f6a66cbbaa260c1cafeccaee185b7b24c5764434f2586760"
Sep 08 16:55:48.717331 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.717295    2557 scope.go:117] "RemoveContainer" containerID="125da4abeaca6239f6a66cbbaa260c1cafeccaee185b7b24c5764434f2586760"
Sep 08 16:55:48.717576 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:55:48.717556    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"125da4abeaca6239f6a66cbbaa260c1cafeccaee185b7b24c5764434f2586760\": container with ID starting with 125da4abeaca6239f6a66cbbaa260c1cafeccaee185b7b24c5764434f2586760 not found: ID does not exist" containerID="125da4abeaca6239f6a66cbbaa260c1cafeccaee185b7b24c5764434f2586760"
Sep 08 16:55:48.717641 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.717585    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"125da4abeaca6239f6a66cbbaa260c1cafeccaee185b7b24c5764434f2586760"} err="failed to get container status \"125da4abeaca6239f6a66cbbaa260c1cafeccaee185b7b24c5764434f2586760\": rpc error: code = NotFound desc = could not find container \"125da4abeaca6239f6a66cbbaa260c1cafeccaee185b7b24c5764434f2586760\": container with ID starting with 125da4abeaca6239f6a66cbbaa260c1cafeccaee185b7b24c5764434f2586760 not found: ID does not exist"
Sep 08 16:55:48.720625 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.720607    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk"
Sep 08 16:55:48.804950 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.804908    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71fadf7d-46e1-4a4a-abc9-8276185c31e6-plugins-config-volume\") pod \"71fadf7d-46e1-4a4a-abc9-8276185c31e6\" (UID: \"71fadf7d-46e1-4a4a-abc9-8276185c31e6\") "
Sep 08 16:55:48.804950 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.804944    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84aaa926-730e-460d-bba8-36bfff099744-plugins-config-volume\") pod \"84aaa926-730e-460d-bba8-36bfff099744\" (UID: \"84aaa926-730e-460d-bba8-36bfff099744\") "
Sep 08 16:55:48.805215 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.804969    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z77x\" (UniqueName: \"kubernetes.io/projected/84aaa926-730e-460d-bba8-36bfff099744-kube-api-access-2z77x\") pod \"84aaa926-730e-460d-bba8-36bfff099744\" (UID: \"84aaa926-730e-460d-bba8-36bfff099744\") "
Sep 08 16:55:48.805215 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.805009    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxnpj\" (UniqueName: \"kubernetes.io/projected/71fadf7d-46e1-4a4a-abc9-8276185c31e6-kube-api-access-sxnpj\") pod \"71fadf7d-46e1-4a4a-abc9-8276185c31e6\" (UID: \"71fadf7d-46e1-4a4a-abc9-8276185c31e6\") "
Sep 08 16:55:48.805390 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.805361    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84aaa926-730e-460d-bba8-36bfff099744-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "84aaa926-730e-460d-bba8-36bfff099744" (UID: "84aaa926-730e-460d-bba8-36bfff099744"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:55:48.805463 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.805361    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71fadf7d-46e1-4a4a-abc9-8276185c31e6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "71fadf7d-46e1-4a4a-abc9-8276185c31e6" (UID: "71fadf7d-46e1-4a4a-abc9-8276185c31e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:55:48.807238 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.807214    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84aaa926-730e-460d-bba8-36bfff099744-kube-api-access-2z77x" (OuterVolumeSpecName: "kube-api-access-2z77x") pod "84aaa926-730e-460d-bba8-36bfff099744" (UID: "84aaa926-730e-460d-bba8-36bfff099744"). InnerVolumeSpecName "kube-api-access-2z77x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:55:48.807359 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.807256    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71fadf7d-46e1-4a4a-abc9-8276185c31e6-kube-api-access-sxnpj" (OuterVolumeSpecName: "kube-api-access-sxnpj") pod "71fadf7d-46e1-4a4a-abc9-8276185c31e6" (UID: "71fadf7d-46e1-4a4a-abc9-8276185c31e6"). InnerVolumeSpecName "kube-api-access-sxnpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:55:48.905609 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.905564    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71fadf7d-46e1-4a4a-abc9-8276185c31e6-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:48.905609 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.905591    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84aaa926-730e-460d-bba8-36bfff099744-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:48.905609 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.905599    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z77x\" (UniqueName: \"kubernetes.io/projected/84aaa926-730e-460d-bba8-36bfff099744-kube-api-access-2z77x\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:48.905609 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.905608    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxnpj\" (UniqueName: \"kubernetes.io/projected/71fadf7d-46e1-4a4a-abc9-8276185c31e6-kube-api-access-sxnpj\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:55:48.944649 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.944562    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e5401-wxnqm"]
Sep 08 16:55:48.950387 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:48.950357    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6e5401-wxnqm"]
Sep 08 16:55:49.706572 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:49.706480    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk"
Sep 08 16:55:49.706572 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:49.706494    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk" event={"ID":"71fadf7d-46e1-4a4a-abc9-8276185c31e6","Type":"ContainerDied","Data":"4865394dfb6d6ef61512c8d49e150b7a5dbc001d49bfd141da7b63e77f854236"}
Sep 08 16:55:49.706572 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:49.706560    2557 scope.go:117] "RemoveContainer" containerID="4815347b806004b749d231e6078cc158b72af08ee93b20e2da8e643d2cd603a5"
Sep 08 16:55:49.708137 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:49.707808    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk"
Sep 08 16:55:49.735842 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:49.735799    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk"]
Sep 08 16:55:49.746946 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:49.746915    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6e5401-657c9b5754-v5nwk"]
Sep 08 16:55:49.761991 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:49.761956    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk"]
Sep 08 16:55:49.768195 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:49.768167    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6e5401-bc65969c-f8clk"]
Sep 08 16:55:50.346467 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:50.346419    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57956374-67bc-48de-8d4d-f155ebef005e" path="/var/lib/kubelet/pods/57956374-67bc-48de-8d4d-f155ebef005e/volumes"
Sep 08 16:55:50.346893 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:50.346872    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71fadf7d-46e1-4a4a-abc9-8276185c31e6" path="/var/lib/kubelet/pods/71fadf7d-46e1-4a4a-abc9-8276185c31e6/volumes"
Sep 08 16:55:50.347268 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:50.347249    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84aaa926-730e-460d-bba8-36bfff099744" path="/var/lib/kubelet/pods/84aaa926-730e-460d-bba8-36bfff099744/volumes"
Sep 08 16:55:53.123237 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:53.123201    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:55:55.540642 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.540612    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r"]
Sep 08 16:55:55.541080 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.541035    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71fadf7d-46e1-4a4a-abc9-8276185c31e6" containerName="payload-pre-processing"
Sep 08 16:55:55.541080 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.541047    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="71fadf7d-46e1-4a4a-abc9-8276185c31e6" containerName="payload-pre-processing"
Sep 08 16:55:55.541080 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.541057    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84aaa926-730e-460d-bba8-36bfff099744" containerName="payload-processing"
Sep 08 16:55:55.541080 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.541065    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="84aaa926-730e-460d-bba8-36bfff099744" containerName="payload-processing"
Sep 08 16:55:55.541080 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.541078    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57956374-67bc-48de-8d4d-f155ebef005e" containerName="revoke-keys"
Sep 08 16:55:55.541080 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.541084    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="57956374-67bc-48de-8d4d-f155ebef005e" containerName="revoke-keys"
Sep 08 16:55:55.541280 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.541163    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="71fadf7d-46e1-4a4a-abc9-8276185c31e6" containerName="payload-pre-processing"
Sep 08 16:55:55.541280 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.541176    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="84aaa926-730e-460d-bba8-36bfff099744" containerName="payload-processing"
Sep 08 16:55:55.541280 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.541184    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="57956374-67bc-48de-8d4d-f155ebef005e" containerName="revoke-keys"
Sep 08 16:55:55.545775 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.545754    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r"
Sep 08 16:55:55.549789 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.549768    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-84ba67\""
Sep 08 16:55:55.551730 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.551707    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-84ba67-dockercfg-gksbc\""
Sep 08 16:55:55.576193 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.576153    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r"]
Sep 08 16:55:55.627825 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.627789    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l"]
Sep 08 16:55:55.635177 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.635048    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l"
Sep 08 16:55:55.667219 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.667180    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bb403a-fb66-4d6a-ae69-57ee740b2d56-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r\" (UID: \"f5bb403a-fb66-4d6a-ae69-57ee740b2d56\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r"
Sep 08 16:55:55.667219 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.667219    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znvt4\" (UniqueName: \"kubernetes.io/projected/f5bb403a-fb66-4d6a-ae69-57ee740b2d56-kube-api-access-znvt4\") pod \"payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r\" (UID: \"f5bb403a-fb66-4d6a-ae69-57ee740b2d56\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r"
Sep 08 16:55:55.691735 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.691696    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l"]
Sep 08 16:55:55.768387 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.768350    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bb403a-fb66-4d6a-ae69-57ee740b2d56-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r\" (UID: \"f5bb403a-fb66-4d6a-ae69-57ee740b2d56\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r"
Sep 08 16:55:55.768387 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.768391    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znvt4\" (UniqueName: \"kubernetes.io/projected/f5bb403a-fb66-4d6a-ae69-57ee740b2d56-kube-api-access-znvt4\") pod \"payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r\" (UID: \"f5bb403a-fb66-4d6a-ae69-57ee740b2d56\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r"
Sep 08 16:55:55.768622 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.768433    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvttn\" (UniqueName: \"kubernetes.io/projected/c7559184-0c46-4686-83fb-1272e0a22579-kube-api-access-mvttn\") pod \"payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l\" (UID: \"c7559184-0c46-4686-83fb-1272e0a22579\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l"
Sep 08 16:55:55.768622 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.768472    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7559184-0c46-4686-83fb-1272e0a22579-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l\" (UID: \"c7559184-0c46-4686-83fb-1272e0a22579\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l"
Sep 08 16:55:55.769165 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.769147    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bb403a-fb66-4d6a-ae69-57ee740b2d56-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r\" (UID: \"f5bb403a-fb66-4d6a-ae69-57ee740b2d56\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r"
Sep 08 16:55:55.813451 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.813369    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znvt4\" (UniqueName: \"kubernetes.io/projected/f5bb403a-fb66-4d6a-ae69-57ee740b2d56-kube-api-access-znvt4\") pod \"payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r\" (UID: \"f5bb403a-fb66-4d6a-ae69-57ee740b2d56\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r"
Sep 08 16:55:55.856270 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.856235    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r"
Sep 08 16:55:55.869660 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.869628    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7559184-0c46-4686-83fb-1272e0a22579-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l\" (UID: \"c7559184-0c46-4686-83fb-1272e0a22579\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l"
Sep 08 16:55:55.869804 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.869720    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvttn\" (UniqueName: \"kubernetes.io/projected/c7559184-0c46-4686-83fb-1272e0a22579-kube-api-access-mvttn\") pod \"payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l\" (UID: \"c7559184-0c46-4686-83fb-1272e0a22579\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l"
Sep 08 16:55:55.870485 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.870459    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7559184-0c46-4686-83fb-1272e0a22579-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l\" (UID: \"c7559184-0c46-4686-83fb-1272e0a22579\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l"
Sep 08 16:55:55.888270 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.888242    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvttn\" (UniqueName: \"kubernetes.io/projected/c7559184-0c46-4686-83fb-1272e0a22579-kube-api-access-mvttn\") pod \"payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l\" (UID: \"c7559184-0c46-4686-83fb-1272e0a22579\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l"
Sep 08 16:55:55.946280 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:55.946247    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l"
Sep 08 16:55:56.185446 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:56.185414    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r"]
Sep 08 16:55:56.187320 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:55:56.187289    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5bb403a_fb66_4d6a_ae69_57ee740b2d56.slice/crio-c0d351dc3ac401146e955747b97b6c5e8a9ddb8851388a38128ad3de7a7e6390 WatchSource:0}: Error finding container c0d351dc3ac401146e955747b97b6c5e8a9ddb8851388a38128ad3de7a7e6390: Status 404 returned error can't find the container with id c0d351dc3ac401146e955747b97b6c5e8a9ddb8851388a38128ad3de7a7e6390
Sep 08 16:55:56.248791 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:56.248766    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l"]
Sep 08 16:55:56.250332 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:55:56.250305    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7559184_0c46_4686_83fb_1272e0a22579.slice/crio-1aadb3ce8d673f0a1ff4de8f68428fd1452b277a3cbe75b708dbc66dd9b7132c WatchSource:0}: Error finding container 1aadb3ce8d673f0a1ff4de8f68428fd1452b277a3cbe75b708dbc66dd9b7132c: Status 404 returned error can't find the container with id 1aadb3ce8d673f0a1ff4de8f68428fd1452b277a3cbe75b708dbc66dd9b7132c
Sep 08 16:55:56.748416 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:56.748383    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l" event={"ID":"c7559184-0c46-4686-83fb-1272e0a22579","Type":"ContainerStarted","Data":"8f0e2308729a185fa84d81989fd2192bfaec4d35003ce3af4aa6ff04984bcf42"}
Sep 08 16:55:56.748904 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:56.748440    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l" event={"ID":"c7559184-0c46-4686-83fb-1272e0a22579","Type":"ContainerStarted","Data":"1aadb3ce8d673f0a1ff4de8f68428fd1452b277a3cbe75b708dbc66dd9b7132c"}
Sep 08 16:55:56.748904 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:56.748613    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l"
Sep 08 16:55:56.750257 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:56.750219    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r" event={"ID":"f5bb403a-fb66-4d6a-ae69-57ee740b2d56","Type":"ContainerStarted","Data":"96664e909c91b1c3064e3431aef640a01992334e01724bf7243930d167f85cd3"}
Sep 08 16:55:56.750403 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:56.750264    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r" event={"ID":"f5bb403a-fb66-4d6a-ae69-57ee740b2d56","Type":"ContainerStarted","Data":"c0d351dc3ac401146e955747b97b6c5e8a9ddb8851388a38128ad3de7a7e6390"}
Sep 08 16:55:56.750403 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:56.750299    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r"
Sep 08 16:55:56.782276 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:56.782223    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l" podStartSLOduration=1.782201416 podStartE2EDuration="1.782201416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:55:56.78173835 +0000 UTC m=+1540.944709797" watchObservedRunningTime="2026-09-08 16:55:56.782201416 +0000 UTC m=+1540.945172858"
Sep 08 16:55:56.863316 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:55:56.863255    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r" podStartSLOduration=1.863238704 podStartE2EDuration="1.863238704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:55:56.840593412 +0000 UTC m=+1541.003564863" watchObservedRunningTime="2026-09-08 16:55:56.863238704 +0000 UTC m=+1541.026210152"
Sep 08 16:56:05.073128 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:05.073093    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:56:07.757725 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:07.757694    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r"
Sep 08 16:56:07.758167 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:07.757823    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l"
Sep 08 16:56:23.229877 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:23.229826    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7vgjf_050c1b6a-d113-4083-adb8-b94cb80b1f5c/payload-processing/0.log"
Sep 08 16:56:23.369255 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:23.369225    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r_f5bb403a-fb66-4d6a-ae69-57ee740b2d56/payload-processing/0.log"
Sep 08 16:56:27.559319 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.559284    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"]
Sep 08 16:56:27.564194 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.564170    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:56:27.567940 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.567914    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d2b1cc\"/\"default-dockercfg-665nj\""
Sep 08 16:56:27.568196 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.568180    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d2b1cc\"/\"openshift-service-ca.crt\""
Sep 08 16:56:27.568304 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.568279    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d2b1cc\"/\"ipp-route-d2b1cc-kserve-self-signed-certs\""
Sep 08 16:56:27.568366 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.568336    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d2b1cc\"/\"kube-root-ca.crt\""
Sep 08 16:56:27.583330 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.583297    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"]
Sep 08 16:56:27.653343 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.653311    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/135ef8c3-562b-4dbf-8864-36b4152c69d7-home\") pod \"ipp-route-d2b1cc-kserve-658db547f9-c5x4d\" (UID: \"135ef8c3-562b-4dbf-8864-36b4152c69d7\") " pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:56:27.653547 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.653365    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/135ef8c3-562b-4dbf-8864-36b4152c69d7-tls-certs\") pod \"ipp-route-d2b1cc-kserve-658db547f9-c5x4d\" (UID: \"135ef8c3-562b-4dbf-8864-36b4152c69d7\") " pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:56:27.653547 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.653387    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/135ef8c3-562b-4dbf-8864-36b4152c69d7-dshm\") pod \"ipp-route-d2b1cc-kserve-658db547f9-c5x4d\" (UID: \"135ef8c3-562b-4dbf-8864-36b4152c69d7\") " pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:56:27.653547 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.653473    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/135ef8c3-562b-4dbf-8864-36b4152c69d7-model-cache\") pod \"ipp-route-d2b1cc-kserve-658db547f9-c5x4d\" (UID: \"135ef8c3-562b-4dbf-8864-36b4152c69d7\") " pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:56:27.653547 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.653504    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/135ef8c3-562b-4dbf-8864-36b4152c69d7-tmp-dir\") pod \"ipp-route-d2b1cc-kserve-658db547f9-c5x4d\" (UID: \"135ef8c3-562b-4dbf-8864-36b4152c69d7\") " pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:56:27.653547 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.653521    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tlnm\" (UniqueName: \"kubernetes.io/projected/135ef8c3-562b-4dbf-8864-36b4152c69d7-kube-api-access-2tlnm\") pod \"ipp-route-d2b1cc-kserve-658db547f9-c5x4d\" (UID: \"135ef8c3-562b-4dbf-8864-36b4152c69d7\") " pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:56:27.754801 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.754756    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/135ef8c3-562b-4dbf-8864-36b4152c69d7-dshm\") pod \"ipp-route-d2b1cc-kserve-658db547f9-c5x4d\" (UID: \"135ef8c3-562b-4dbf-8864-36b4152c69d7\") " pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:56:27.754801 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.754803    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/135ef8c3-562b-4dbf-8864-36b4152c69d7-model-cache\") pod \"ipp-route-d2b1cc-kserve-658db547f9-c5x4d\" (UID: \"135ef8c3-562b-4dbf-8864-36b4152c69d7\") " pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:56:27.755093 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.754828    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/135ef8c3-562b-4dbf-8864-36b4152c69d7-tmp-dir\") pod \"ipp-route-d2b1cc-kserve-658db547f9-c5x4d\" (UID: \"135ef8c3-562b-4dbf-8864-36b4152c69d7\") " pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:56:27.755093 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.754882    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tlnm\" (UniqueName: \"kubernetes.io/projected/135ef8c3-562b-4dbf-8864-36b4152c69d7-kube-api-access-2tlnm\") pod \"ipp-route-d2b1cc-kserve-658db547f9-c5x4d\" (UID: \"135ef8c3-562b-4dbf-8864-36b4152c69d7\") " pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:56:27.755093 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.754963    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/135ef8c3-562b-4dbf-8864-36b4152c69d7-home\") pod \"ipp-route-d2b1cc-kserve-658db547f9-c5x4d\" (UID: \"135ef8c3-562b-4dbf-8864-36b4152c69d7\") " pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:56:27.755093 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.755031    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/135ef8c3-562b-4dbf-8864-36b4152c69d7-tls-certs\") pod \"ipp-route-d2b1cc-kserve-658db547f9-c5x4d\" (UID: \"135ef8c3-562b-4dbf-8864-36b4152c69d7\") " pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:56:27.755289 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.755261    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/135ef8c3-562b-4dbf-8864-36b4152c69d7-tmp-dir\") pod \"ipp-route-d2b1cc-kserve-658db547f9-c5x4d\" (UID: \"135ef8c3-562b-4dbf-8864-36b4152c69d7\") " pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:56:27.755357 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.755339    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/135ef8c3-562b-4dbf-8864-36b4152c69d7-model-cache\") pod \"ipp-route-d2b1cc-kserve-658db547f9-c5x4d\" (UID: \"135ef8c3-562b-4dbf-8864-36b4152c69d7\") " pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:56:27.755419 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.755399    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/135ef8c3-562b-4dbf-8864-36b4152c69d7-home\") pod \"ipp-route-d2b1cc-kserve-658db547f9-c5x4d\" (UID: \"135ef8c3-562b-4dbf-8864-36b4152c69d7\") " pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:56:27.757373 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.757338    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/135ef8c3-562b-4dbf-8864-36b4152c69d7-dshm\") pod \"ipp-route-d2b1cc-kserve-658db547f9-c5x4d\" (UID: \"135ef8c3-562b-4dbf-8864-36b4152c69d7\") " pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:56:27.757611 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.757585    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/135ef8c3-562b-4dbf-8864-36b4152c69d7-tls-certs\") pod \"ipp-route-d2b1cc-kserve-658db547f9-c5x4d\" (UID: \"135ef8c3-562b-4dbf-8864-36b4152c69d7\") " pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:56:27.768787 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.768759    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tlnm\" (UniqueName: \"kubernetes.io/projected/135ef8c3-562b-4dbf-8864-36b4152c69d7-kube-api-access-2tlnm\") pod \"ipp-route-d2b1cc-kserve-658db547f9-c5x4d\" (UID: \"135ef8c3-562b-4dbf-8864-36b4152c69d7\") " pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:56:27.873804 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:27.873711    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:56:28.158149 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:28.158052    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"]
Sep 08 16:56:28.160846 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:56:28.160804    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod135ef8c3_562b_4dbf_8864_36b4152c69d7.slice/crio-b78003245f45b72bcecabca8bb576f0d2675d216c3915e5177482b02e4711f29 WatchSource:0}: Error finding container b78003245f45b72bcecabca8bb576f0d2675d216c3915e5177482b02e4711f29: Status 404 returned error can't find the container with id b78003245f45b72bcecabca8bb576f0d2675d216c3915e5177482b02e4711f29
Sep 08 16:56:28.917974 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:28.917940    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d" event={"ID":"135ef8c3-562b-4dbf-8864-36b4152c69d7","Type":"ContainerStarted","Data":"5f2a7a8fac8f88a25775d72a08ef36925e88790b328afe49295486f79e706e25"}
Sep 08 16:56:28.918477 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:28.917987    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d" event={"ID":"135ef8c3-562b-4dbf-8864-36b4152c69d7","Type":"ContainerStarted","Data":"b78003245f45b72bcecabca8bb576f0d2675d216c3915e5177482b02e4711f29"}
Sep 08 16:56:28.954059 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:28.954008    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d" podStartSLOduration=1.953989875 podStartE2EDuration="1.953989875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:56:28.952905095 +0000 UTC m=+1573.115876542" watchObservedRunningTime="2026-09-08 16:56:28.953989875 +0000 UTC m=+1573.116961323"
Sep 08 16:56:31.392895 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:31.392856    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:56:37.874591 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:37.874551    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:56:37.875111 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:37.874614    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:56:37.887982 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:37.887953    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:56:37.976941 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:37.976909    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:56:42.744790 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:42.744750    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:56:48.839405 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:48.839374    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf_c2189b15-dc78-4597-a0ff-1bec9bf395ad/payload-processing/0.log"
Sep 08 16:56:48.973972 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:48.973939    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7vgjf_050c1b6a-d113-4083-adb8-b94cb80b1f5c/payload-processing/0.log"
Sep 08 16:56:49.094903 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:49.094814    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r_f5bb403a-fb66-4d6a-ae69-57ee740b2d56/payload-processing/0.log"
Sep 08 16:56:55.704974 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:55.704937    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv"]
Sep 08 16:56:55.708184 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:55.708166    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv"
Sep 08 16:56:55.711795 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:55.711774    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-a0a70f\""
Sep 08 16:56:55.711931 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:55.711870    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-a0a70f-dockercfg-4psn8\""
Sep 08 16:56:55.726500 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:55.726464    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv"]
Sep 08 16:56:55.776307 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:55.776272    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6"]
Sep 08 16:56:55.780272 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:55.780253    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6"
Sep 08 16:56:55.798225 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:55.798197    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6"]
Sep 08 16:56:55.809047 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:55.809011    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxp94\" (UniqueName: \"kubernetes.io/projected/6e375cca-c8d0-4b85-b0e4-11299375272b-kube-api-access-rxp94\") pod \"payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv\" (UID: \"6e375cca-c8d0-4b85-b0e4-11299375272b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv"
Sep 08 16:56:55.809239 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:55.809109    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e375cca-c8d0-4b85-b0e4-11299375272b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv\" (UID: \"6e375cca-c8d0-4b85-b0e4-11299375272b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv"
Sep 08 16:56:55.909904 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:55.909864    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e375cca-c8d0-4b85-b0e4-11299375272b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv\" (UID: \"6e375cca-c8d0-4b85-b0e4-11299375272b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv"
Sep 08 16:56:55.910110 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:55.909943    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxp94\" (UniqueName: \"kubernetes.io/projected/6e375cca-c8d0-4b85-b0e4-11299375272b-kube-api-access-rxp94\") pod \"payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv\" (UID: \"6e375cca-c8d0-4b85-b0e4-11299375272b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv"
Sep 08 16:56:55.910110 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:55.909987    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5179bbc4-f4ab-46c3-b557-138b3d03f0bc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6\" (UID: \"5179bbc4-f4ab-46c3-b557-138b3d03f0bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6"
Sep 08 16:56:55.910110 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:55.910005    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8q5m\" (UniqueName: \"kubernetes.io/projected/5179bbc4-f4ab-46c3-b557-138b3d03f0bc-kube-api-access-q8q5m\") pod \"payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6\" (UID: \"5179bbc4-f4ab-46c3-b557-138b3d03f0bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6"
Sep 08 16:56:55.910631 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:55.910608    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e375cca-c8d0-4b85-b0e4-11299375272b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv\" (UID: \"6e375cca-c8d0-4b85-b0e4-11299375272b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv"
Sep 08 16:56:55.921490 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:55.921453    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxp94\" (UniqueName: \"kubernetes.io/projected/6e375cca-c8d0-4b85-b0e4-11299375272b-kube-api-access-rxp94\") pod \"payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv\" (UID: \"6e375cca-c8d0-4b85-b0e4-11299375272b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv"
Sep 08 16:56:56.010555 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:56.010455    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5179bbc4-f4ab-46c3-b557-138b3d03f0bc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6\" (UID: \"5179bbc4-f4ab-46c3-b557-138b3d03f0bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6"
Sep 08 16:56:56.010555 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:56.010499    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8q5m\" (UniqueName: \"kubernetes.io/projected/5179bbc4-f4ab-46c3-b557-138b3d03f0bc-kube-api-access-q8q5m\") pod \"payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6\" (UID: \"5179bbc4-f4ab-46c3-b557-138b3d03f0bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6"
Sep 08 16:56:56.011273 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:56.011252    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5179bbc4-f4ab-46c3-b557-138b3d03f0bc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6\" (UID: \"5179bbc4-f4ab-46c3-b557-138b3d03f0bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6"
Sep 08 16:56:56.019399 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:56.019358    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv"
Sep 08 16:56:56.022197 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:56.022171    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8q5m\" (UniqueName: \"kubernetes.io/projected/5179bbc4-f4ab-46c3-b557-138b3d03f0bc-kube-api-access-q8q5m\") pod \"payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6\" (UID: \"5179bbc4-f4ab-46c3-b557-138b3d03f0bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6"
Sep 08 16:56:56.091349 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:56.091316    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6"
Sep 08 16:56:56.322123 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:56.322085    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv"]
Sep 08 16:56:56.322682 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:56:56.322656    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e375cca_c8d0_4b85_b0e4_11299375272b.slice/crio-2a1a4249a5b7f22b30d88345aa10101de8bbceba7de0652ecf0f3e23efd2e445 WatchSource:0}: Error finding container 2a1a4249a5b7f22b30d88345aa10101de8bbceba7de0652ecf0f3e23efd2e445: Status 404 returned error can't find the container with id 2a1a4249a5b7f22b30d88345aa10101de8bbceba7de0652ecf0f3e23efd2e445
Sep 08 16:56:56.375223 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:56.375193    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6"]
Sep 08 16:56:56.377089 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:56:56.377057    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5179bbc4_f4ab_46c3_b557_138b3d03f0bc.slice/crio-b640b079e687352d4967535aa60e043da7fc2dd5c4e9e0367966980973eb4977 WatchSource:0}: Error finding container b640b079e687352d4967535aa60e043da7fc2dd5c4e9e0367966980973eb4977: Status 404 returned error can't find the container with id b640b079e687352d4967535aa60e043da7fc2dd5c4e9e0367966980973eb4977
Sep 08 16:56:57.064575 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:57.064539    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6" event={"ID":"5179bbc4-f4ab-46c3-b557-138b3d03f0bc","Type":"ContainerStarted","Data":"1ae10956c9c8356152f73ba9f7118e7482e60377cfa060157f661759130f7644"}
Sep 08 16:56:57.065066 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:57.064598    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6" event={"ID":"5179bbc4-f4ab-46c3-b557-138b3d03f0bc","Type":"ContainerStarted","Data":"b640b079e687352d4967535aa60e043da7fc2dd5c4e9e0367966980973eb4977"}
Sep 08 16:56:57.065066 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:57.064705    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6"
Sep 08 16:56:57.066326 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:57.066297    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv" event={"ID":"6e375cca-c8d0-4b85-b0e4-11299375272b","Type":"ContainerStarted","Data":"2967c507bea79048235d556d3e776eaf26224b6e9529662b980a88950629caeb"}
Sep 08 16:56:57.066490 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:57.066346    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv" event={"ID":"6e375cca-c8d0-4b85-b0e4-11299375272b","Type":"ContainerStarted","Data":"2a1a4249a5b7f22b30d88345aa10101de8bbceba7de0652ecf0f3e23efd2e445"}
Sep 08 16:56:57.066490 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:57.066399    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv"
Sep 08 16:56:57.214322 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:57.214263    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv" podStartSLOduration=2.214240988 podStartE2EDuration="2.214240988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:56:57.211696188 +0000 UTC m=+1601.374667636" watchObservedRunningTime="2026-09-08 16:56:57.214240988 +0000 UTC m=+1601.377212436"
Sep 08 16:56:57.214977 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:56:57.214937    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6" podStartSLOduration=2.214921872 podStartE2EDuration="2.214921872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:56:57.129098906 +0000 UTC m=+1601.292070358" watchObservedRunningTime="2026-09-08 16:56:57.214921872 +0000 UTC m=+1601.377893324"
Sep 08 16:57:01.284986 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:01.284946    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:57:08.075596 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:08.075509    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6"
Sep 08 16:57:08.076072 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:08.075811    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv"
Sep 08 16:57:16.711577 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:16.711546    2557 scope.go:117] "RemoveContainer" containerID="b5db37f83c772e7d07b89b1664c9bae03aaff1c708d4de762848f97c6d4271da"
Sep 08 16:57:19.165702 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:19.165663    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a0a70f-b7gtg"]
Sep 08 16:57:19.170398 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:19.170378    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a0a70f-b7gtg"
Sep 08 16:57:19.173867 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:19.173829    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-886jf\""
Sep 08 16:57:19.174819 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:19.174802    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:57:19.189102 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:19.189063    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a0a70f-b7gtg"]
Sep 08 16:57:19.206327 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:19.206294    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f818ab0-53fb-40ca-8ad2-e796550fdb8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a0a70f-b7gtg\" (UID: \"9f818ab0-53fb-40ca-8ad2-e796550fdb8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a0a70f-b7gtg"
Sep 08 16:57:19.307673 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:19.307630    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f818ab0-53fb-40ca-8ad2-e796550fdb8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a0a70f-b7gtg\" (UID: \"9f818ab0-53fb-40ca-8ad2-e796550fdb8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a0a70f-b7gtg"
Sep 08 16:57:19.308361 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:19.308342    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f818ab0-53fb-40ca-8ad2-e796550fdb8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a0a70f-b7gtg\" (UID: \"9f818ab0-53fb-40ca-8ad2-e796550fdb8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a0a70f-b7gtg"
Sep 08 16:57:19.481534 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:19.481443    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a0a70f-b7gtg"
Sep 08 16:57:19.794629 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:19.794602    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a0a70f-b7gtg"]
Sep 08 16:57:19.796020 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:57:19.795983    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f818ab0_53fb_40ca_8ad2_e796550fdb8d.slice/crio-4a850b334abea5110e296790311b26f72f41b42ecf11d395cc2da5118189fac1 WatchSource:0}: Error finding container 4a850b334abea5110e296790311b26f72f41b42ecf11d395cc2da5118189fac1: Status 404 returned error can't find the container with id 4a850b334abea5110e296790311b26f72f41b42ecf11d395cc2da5118189fac1
Sep 08 16:57:20.190679 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:20.190642    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a0a70f-b7gtg" event={"ID":"9f818ab0-53fb-40ca-8ad2-e796550fdb8d","Type":"ContainerStarted","Data":"0b92a7efa0c5664768e606e1083b94ea2c46f6afd2ed3de9ca4fa5fc2966a277"}
Sep 08 16:57:20.191093 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:20.190698    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a0a70f-b7gtg" event={"ID":"9f818ab0-53fb-40ca-8ad2-e796550fdb8d","Type":"ContainerStarted","Data":"4a850b334abea5110e296790311b26f72f41b42ecf11d395cc2da5118189fac1"}
Sep 08 16:57:20.231273 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:20.231220    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a0a70f-b7gtg" podStartSLOduration=1.23120158 podStartE2EDuration="1.23120158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:57:20.220747883 +0000 UTC m=+1624.383719334" watchObservedRunningTime="2026-09-08 16:57:20.23120158 +0000 UTC m=+1624.394173027"
Sep 08 16:57:21.197210 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:21.197171    2557 generic.go:356] "Generic (PLEG): container finished" podID="9f818ab0-53fb-40ca-8ad2-e796550fdb8d" containerID="0b92a7efa0c5664768e606e1083b94ea2c46f6afd2ed3de9ca4fa5fc2966a277" exitCode=0
Sep 08 16:57:21.197567 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:21.197247    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a0a70f-b7gtg" event={"ID":"9f818ab0-53fb-40ca-8ad2-e796550fdb8d","Type":"ContainerDied","Data":"0b92a7efa0c5664768e606e1083b94ea2c46f6afd2ed3de9ca4fa5fc2966a277"}
Sep 08 16:57:22.344665 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:22.344633    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a0a70f-b7gtg"
Sep 08 16:57:22.435556 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:22.435516    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f818ab0-53fb-40ca-8ad2-e796550fdb8d-maas-api-service-ca\") pod \"9f818ab0-53fb-40ca-8ad2-e796550fdb8d\" (UID: \"9f818ab0-53fb-40ca-8ad2-e796550fdb8d\") "
Sep 08 16:57:22.435960 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:22.435932    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f818ab0-53fb-40ca-8ad2-e796550fdb8d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9f818ab0-53fb-40ca-8ad2-e796550fdb8d" (UID: "9f818ab0-53fb-40ca-8ad2-e796550fdb8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:57:22.536380 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:22.536274    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f818ab0-53fb-40ca-8ad2-e796550fdb8d-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:57:23.208476 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:23.208445    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a0a70f-b7gtg"
Sep 08 16:57:23.208476 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:23.208474    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a0a70f-b7gtg" event={"ID":"9f818ab0-53fb-40ca-8ad2-e796550fdb8d","Type":"ContainerDied","Data":"4a850b334abea5110e296790311b26f72f41b42ecf11d395cc2da5118189fac1"}
Sep 08 16:57:23.208731 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:23.208518    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a850b334abea5110e296790311b26f72f41b42ecf11d395cc2da5118189fac1"
Sep 08 16:57:25.956434 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:25.956390    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6"]
Sep 08 16:57:25.956959 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:25.956616    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6" podUID="5179bbc4-f4ab-46c3-b557-138b3d03f0bc" containerName="payload-processing" containerID="cri-o://1ae10956c9c8356152f73ba9f7118e7482e60377cfa060157f661759130f7644" gracePeriod=30
Sep 08 16:57:25.982976 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:25.982942    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv"]
Sep 08 16:57:25.983218 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:25.983184    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv" podUID="6e375cca-c8d0-4b85-b0e4-11299375272b" containerName="payload-pre-processing" containerID="cri-o://2967c507bea79048235d556d3e776eaf26224b6e9529662b980a88950629caeb" gracePeriod=30
Sep 08 16:57:26.238662 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:26.238560    2557 generic.go:356] "Generic (PLEG): container finished" podID="5179bbc4-f4ab-46c3-b557-138b3d03f0bc" containerID="1ae10956c9c8356152f73ba9f7118e7482e60377cfa060157f661759130f7644" exitCode=0
Sep 08 16:57:26.238662 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:26.238636    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6" event={"ID":"5179bbc4-f4ab-46c3-b557-138b3d03f0bc","Type":"ContainerDied","Data":"1ae10956c9c8356152f73ba9f7118e7482e60377cfa060157f661759130f7644"}
Sep 08 16:57:26.240442 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:26.240415    2557 generic.go:356] "Generic (PLEG): container finished" podID="6e375cca-c8d0-4b85-b0e4-11299375272b" containerID="2967c507bea79048235d556d3e776eaf26224b6e9529662b980a88950629caeb" exitCode=0
Sep 08 16:57:26.240551 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:26.240495    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv" event={"ID":"6e375cca-c8d0-4b85-b0e4-11299375272b","Type":"ContainerDied","Data":"2967c507bea79048235d556d3e776eaf26224b6e9529662b980a88950629caeb"}
Sep 08 16:57:26.272613 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:26.272549    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6"
Sep 08 16:57:26.276704 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:26.276654    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv"
Sep 08 16:57:26.370814 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:26.370765    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxp94\" (UniqueName: \"kubernetes.io/projected/6e375cca-c8d0-4b85-b0e4-11299375272b-kube-api-access-rxp94\") pod \"6e375cca-c8d0-4b85-b0e4-11299375272b\" (UID: \"6e375cca-c8d0-4b85-b0e4-11299375272b\") "
Sep 08 16:57:26.370814 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:26.370813    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e375cca-c8d0-4b85-b0e4-11299375272b-plugins-config-volume\") pod \"6e375cca-c8d0-4b85-b0e4-11299375272b\" (UID: \"6e375cca-c8d0-4b85-b0e4-11299375272b\") "
Sep 08 16:57:26.371086 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:26.370906    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8q5m\" (UniqueName: \"kubernetes.io/projected/5179bbc4-f4ab-46c3-b557-138b3d03f0bc-kube-api-access-q8q5m\") pod \"5179bbc4-f4ab-46c3-b557-138b3d03f0bc\" (UID: \"5179bbc4-f4ab-46c3-b557-138b3d03f0bc\") "
Sep 08 16:57:26.371086 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:26.370948    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5179bbc4-f4ab-46c3-b557-138b3d03f0bc-plugins-config-volume\") pod \"5179bbc4-f4ab-46c3-b557-138b3d03f0bc\" (UID: \"5179bbc4-f4ab-46c3-b557-138b3d03f0bc\") "
Sep 08 16:57:26.371633 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:26.371603    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5179bbc4-f4ab-46c3-b557-138b3d03f0bc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5179bbc4-f4ab-46c3-b557-138b3d03f0bc" (UID: "5179bbc4-f4ab-46c3-b557-138b3d03f0bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:57:26.371778 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:26.371689    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e375cca-c8d0-4b85-b0e4-11299375272b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e375cca-c8d0-4b85-b0e4-11299375272b" (UID: "6e375cca-c8d0-4b85-b0e4-11299375272b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:57:26.374265 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:26.374044    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e375cca-c8d0-4b85-b0e4-11299375272b-kube-api-access-rxp94" (OuterVolumeSpecName: "kube-api-access-rxp94") pod "6e375cca-c8d0-4b85-b0e4-11299375272b" (UID: "6e375cca-c8d0-4b85-b0e4-11299375272b"). InnerVolumeSpecName "kube-api-access-rxp94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:57:26.374265 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:26.374153    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5179bbc4-f4ab-46c3-b557-138b3d03f0bc-kube-api-access-q8q5m" (OuterVolumeSpecName: "kube-api-access-q8q5m") pod "5179bbc4-f4ab-46c3-b557-138b3d03f0bc" (UID: "5179bbc4-f4ab-46c3-b557-138b3d03f0bc"). InnerVolumeSpecName "kube-api-access-q8q5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:57:26.458450 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:26.458398    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a0a70f-b7gtg"]
Sep 08 16:57:26.460089 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:26.460066    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a0a70f-b7gtg"]
Sep 08 16:57:26.471853 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:26.471797    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxp94\" (UniqueName: \"kubernetes.io/projected/6e375cca-c8d0-4b85-b0e4-11299375272b-kube-api-access-rxp94\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:57:26.471853 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:26.471860    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e375cca-c8d0-4b85-b0e4-11299375272b-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:57:26.472106 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:26.471878    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8q5m\" (UniqueName: \"kubernetes.io/projected/5179bbc4-f4ab-46c3-b557-138b3d03f0bc-kube-api-access-q8q5m\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:57:26.472106 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:26.471892    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5179bbc4-f4ab-46c3-b557-138b3d03f0bc-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:57:27.247456 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:27.247427    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6"
Sep 08 16:57:27.247456 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:27.247445    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6" event={"ID":"5179bbc4-f4ab-46c3-b557-138b3d03f0bc","Type":"ContainerDied","Data":"b640b079e687352d4967535aa60e043da7fc2dd5c4e9e0367966980973eb4977"}
Sep 08 16:57:27.248012 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:27.247509    2557 scope.go:117] "RemoveContainer" containerID="1ae10956c9c8356152f73ba9f7118e7482e60377cfa060157f661759130f7644"
Sep 08 16:57:27.248944 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:27.248916    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv" event={"ID":"6e375cca-c8d0-4b85-b0e4-11299375272b","Type":"ContainerDied","Data":"2a1a4249a5b7f22b30d88345aa10101de8bbceba7de0652ecf0f3e23efd2e445"}
Sep 08 16:57:27.248944 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:27.248930    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv"
Sep 08 16:57:27.257145 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:27.257126    2557 scope.go:117] "RemoveContainer" containerID="2967c507bea79048235d556d3e776eaf26224b6e9529662b980a88950629caeb"
Sep 08 16:57:27.290128 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:27.290094    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6"]
Sep 08 16:57:27.305937 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:27.305904    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a0a70f-66897c9c45-4g6l6"]
Sep 08 16:57:27.340131 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:27.340096    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv"]
Sep 08 16:57:27.350137 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:27.350105    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a0a70f-79d76567db-97xlv"]
Sep 08 16:57:28.345770 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:28.345737    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5179bbc4-f4ab-46c3-b557-138b3d03f0bc" path="/var/lib/kubelet/pods/5179bbc4-f4ab-46c3-b557-138b3d03f0bc/volumes"
Sep 08 16:57:28.346225 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:28.346209    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e375cca-c8d0-4b85-b0e4-11299375272b" path="/var/lib/kubelet/pods/6e375cca-c8d0-4b85-b0e4-11299375272b/volumes"
Sep 08 16:57:28.346510 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:28.346498    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f818ab0-53fb-40ca-8ad2-e796550fdb8d" path="/var/lib/kubelet/pods/9f818ab0-53fb-40ca-8ad2-e796550fdb8d/volumes"
Sep 08 16:57:30.513073 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:30.513035    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:57:38.720646 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:38.720612    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2b1cc-5cdpl"]
Sep 08 16:57:38.721044 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:38.721027    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f818ab0-53fb-40ca-8ad2-e796550fdb8d" containerName="revoke-keys"
Sep 08 16:57:38.721044 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:38.721040    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f818ab0-53fb-40ca-8ad2-e796550fdb8d" containerName="revoke-keys"
Sep 08 16:57:38.721119 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:38.721051    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e375cca-c8d0-4b85-b0e4-11299375272b" containerName="payload-pre-processing"
Sep 08 16:57:38.721119 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:38.721058    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e375cca-c8d0-4b85-b0e4-11299375272b" containerName="payload-pre-processing"
Sep 08 16:57:38.721119 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:38.721064    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5179bbc4-f4ab-46c3-b557-138b3d03f0bc" containerName="payload-processing"
Sep 08 16:57:38.721119 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:38.721069    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5179bbc4-f4ab-46c3-b557-138b3d03f0bc" containerName="payload-processing"
Sep 08 16:57:38.721237 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:38.721122    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5179bbc4-f4ab-46c3-b557-138b3d03f0bc" containerName="payload-processing"
Sep 08 16:57:38.721237 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:38.721133    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e375cca-c8d0-4b85-b0e4-11299375272b" containerName="payload-pre-processing"
Sep 08 16:57:38.721237 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:38.721141    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f818ab0-53fb-40ca-8ad2-e796550fdb8d" containerName="revoke-keys"
Sep 08 16:57:38.725754 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:38.725738    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2b1cc-5cdpl"
Sep 08 16:57:38.730580 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:38.730564    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-886jf\""
Sep 08 16:57:38.730824 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:38.730806    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:57:38.748060 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:38.748036    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2b1cc-5cdpl"]
Sep 08 16:57:38.786561 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:38.786516    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5b59a4e-8704-4a98-8d12-a4afe8d1e126-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d2b1cc-5cdpl\" (UID: \"a5b59a4e-8704-4a98-8d12-a4afe8d1e126\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2b1cc-5cdpl"
Sep 08 16:57:38.887255 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:38.887213    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5b59a4e-8704-4a98-8d12-a4afe8d1e126-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d2b1cc-5cdpl\" (UID: \"a5b59a4e-8704-4a98-8d12-a4afe8d1e126\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2b1cc-5cdpl"
Sep 08 16:57:38.887848 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:38.887810    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5b59a4e-8704-4a98-8d12-a4afe8d1e126-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d2b1cc-5cdpl\" (UID: \"a5b59a4e-8704-4a98-8d12-a4afe8d1e126\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2b1cc-5cdpl"
Sep 08 16:57:39.035358 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:39.035255    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2b1cc-5cdpl"
Sep 08 16:57:39.326226 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:39.326197    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2b1cc-5cdpl"]
Sep 08 16:57:39.327969 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:57:39.327936    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b59a4e_8704_4a98_8d12_a4afe8d1e126.slice/crio-3b958ca6f13b577273ddc24e7097b22ae053f44d4251f9858c76079571791254 WatchSource:0}: Error finding container 3b958ca6f13b577273ddc24e7097b22ae053f44d4251f9858c76079571791254: Status 404 returned error can't find the container with id 3b958ca6f13b577273ddc24e7097b22ae053f44d4251f9858c76079571791254
Sep 08 16:57:40.320328 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:40.320236    2557 generic.go:356] "Generic (PLEG): container finished" podID="a5b59a4e-8704-4a98-8d12-a4afe8d1e126" containerID="39cd260d4fc87f0348746424a03a1e4a575fe4ebbd1c28716189b77c4eb3dc2b" exitCode=0
Sep 08 16:57:40.320675 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:40.320325    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2b1cc-5cdpl" event={"ID":"a5b59a4e-8704-4a98-8d12-a4afe8d1e126","Type":"ContainerDied","Data":"39cd260d4fc87f0348746424a03a1e4a575fe4ebbd1c28716189b77c4eb3dc2b"}
Sep 08 16:57:40.320675 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:40.320372    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2b1cc-5cdpl" event={"ID":"a5b59a4e-8704-4a98-8d12-a4afe8d1e126","Type":"ContainerStarted","Data":"3b958ca6f13b577273ddc24e7097b22ae053f44d4251f9858c76079571791254"}
Sep 08 16:57:41.465345 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:41.465316    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2b1cc-5cdpl"
Sep 08 16:57:41.610656 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:41.610565    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5b59a4e-8704-4a98-8d12-a4afe8d1e126-maas-api-service-ca\") pod \"a5b59a4e-8704-4a98-8d12-a4afe8d1e126\" (UID: \"a5b59a4e-8704-4a98-8d12-a4afe8d1e126\") "
Sep 08 16:57:41.611011 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:41.610978    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b59a4e-8704-4a98-8d12-a4afe8d1e126-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a5b59a4e-8704-4a98-8d12-a4afe8d1e126" (UID: "a5b59a4e-8704-4a98-8d12-a4afe8d1e126"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:57:41.711862 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:41.711792    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5b59a4e-8704-4a98-8d12-a4afe8d1e126-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:57:42.332120 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:42.332088    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2b1cc-5cdpl"
Sep 08 16:57:42.332298 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:42.332121    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2b1cc-5cdpl" event={"ID":"a5b59a4e-8704-4a98-8d12-a4afe8d1e126","Type":"ContainerDied","Data":"3b958ca6f13b577273ddc24e7097b22ae053f44d4251f9858c76079571791254"}
Sep 08 16:57:42.332298 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:42.332198    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b958ca6f13b577273ddc24e7097b22ae053f44d4251f9858c76079571791254"
Sep 08 16:57:52.458018 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:52.457974    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:57:54.660112 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:54.660076    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf"]
Sep 08 16:57:54.660520 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:54.660338    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf" podUID="c2189b15-dc78-4597-a0ff-1bec9bf395ad" containerName="payload-processing" containerID="cri-o://736461633af7ad8fb707f25139b3c74c27b096236352798081996584148a52ef" gracePeriod=30
Sep 08 16:57:54.927791 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:54.927763    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf"
Sep 08 16:57:55.021572 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:55.021532    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2b1cc-5cdpl"]
Sep 08 16:57:55.026214 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:55.026185    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d2b1cc-5cdpl"]
Sep 08 16:57:55.032354 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:55.032327    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2189b15-dc78-4597-a0ff-1bec9bf395ad-plugins-config-volume\") pod \"c2189b15-dc78-4597-a0ff-1bec9bf395ad\" (UID: \"c2189b15-dc78-4597-a0ff-1bec9bf395ad\") "
Sep 08 16:57:55.032483 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:55.032453    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmmg9\" (UniqueName: \"kubernetes.io/projected/c2189b15-dc78-4597-a0ff-1bec9bf395ad-kube-api-access-bmmg9\") pod \"c2189b15-dc78-4597-a0ff-1bec9bf395ad\" (UID: \"c2189b15-dc78-4597-a0ff-1bec9bf395ad\") "
Sep 08 16:57:55.032714 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:55.032691    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2189b15-dc78-4597-a0ff-1bec9bf395ad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c2189b15-dc78-4597-a0ff-1bec9bf395ad" (UID: "c2189b15-dc78-4597-a0ff-1bec9bf395ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:57:55.034755 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:55.034732    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2189b15-dc78-4597-a0ff-1bec9bf395ad-kube-api-access-bmmg9" (OuterVolumeSpecName: "kube-api-access-bmmg9") pod "c2189b15-dc78-4597-a0ff-1bec9bf395ad" (UID: "c2189b15-dc78-4597-a0ff-1bec9bf395ad"). InnerVolumeSpecName "kube-api-access-bmmg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:57:55.133528 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:55.133474    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmmg9\" (UniqueName: \"kubernetes.io/projected/c2189b15-dc78-4597-a0ff-1bec9bf395ad-kube-api-access-bmmg9\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:57:55.133528 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:55.133521    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2189b15-dc78-4597-a0ff-1bec9bf395ad-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:57:55.401583 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:55.401491    2557 generic.go:356] "Generic (PLEG): container finished" podID="c2189b15-dc78-4597-a0ff-1bec9bf395ad" containerID="736461633af7ad8fb707f25139b3c74c27b096236352798081996584148a52ef" exitCode=0
Sep 08 16:57:55.401583 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:55.401540    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf"
Sep 08 16:57:55.401823 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:55.401584    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf" event={"ID":"c2189b15-dc78-4597-a0ff-1bec9bf395ad","Type":"ContainerDied","Data":"736461633af7ad8fb707f25139b3c74c27b096236352798081996584148a52ef"}
Sep 08 16:57:55.401823 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:55.401645    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf" event={"ID":"c2189b15-dc78-4597-a0ff-1bec9bf395ad","Type":"ContainerDied","Data":"339398cc7c2cddafcbcab9e2868fffb38d7665337b35856a98a0bf4557bda853"}
Sep 08 16:57:55.401823 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:55.401674    2557 scope.go:117] "RemoveContainer" containerID="736461633af7ad8fb707f25139b3c74c27b096236352798081996584148a52ef"
Sep 08 16:57:55.411357 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:55.411333    2557 scope.go:117] "RemoveContainer" containerID="736461633af7ad8fb707f25139b3c74c27b096236352798081996584148a52ef"
Sep 08 16:57:55.411675 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:57:55.411655    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"736461633af7ad8fb707f25139b3c74c27b096236352798081996584148a52ef\": container with ID starting with 736461633af7ad8fb707f25139b3c74c27b096236352798081996584148a52ef not found: ID does not exist" containerID="736461633af7ad8fb707f25139b3c74c27b096236352798081996584148a52ef"
Sep 08 16:57:55.411721 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:55.411685    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"736461633af7ad8fb707f25139b3c74c27b096236352798081996584148a52ef"} err="failed to get container status \"736461633af7ad8fb707f25139b3c74c27b096236352798081996584148a52ef\": rpc error: code = NotFound desc = could not find container \"736461633af7ad8fb707f25139b3c74c27b096236352798081996584148a52ef\": container with ID starting with 736461633af7ad8fb707f25139b3c74c27b096236352798081996584148a52ef not found: ID does not exist"
Sep 08 16:57:55.428419 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:55.428388    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf"]
Sep 08 16:57:55.431971 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:55.431947    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d2b1cc-7b57947b8c-brbcf"]
Sep 08 16:57:56.345386 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:56.345353    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5b59a4e-8704-4a98-8d12-a4afe8d1e126" path="/var/lib/kubelet/pods/a5b59a4e-8704-4a98-8d12-a4afe8d1e126/volumes"
Sep 08 16:57:56.345756 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:57:56.345680    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2189b15-dc78-4597-a0ff-1bec9bf395ad" path="/var/lib/kubelet/pods/c2189b15-dc78-4597-a0ff-1bec9bf395ad/volumes"
Sep 08 16:58:00.355427 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:00.355390    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"]
Sep 08 16:58:05.449207 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:05.449169    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:58:15.378506 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:15.378462    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d" podUID="135ef8c3-562b-4dbf-8864-36b4152c69d7" containerName="main" containerID="cri-o://5f2a7a8fac8f88a25775d72a08ef36925e88790b328afe49295486f79e706e25" gracePeriod=45
Sep 08 16:58:15.513509 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:15.513474    2557 generic.go:356] "Generic (PLEG): container finished" podID="135ef8c3-562b-4dbf-8864-36b4152c69d7" containerID="5f2a7a8fac8f88a25775d72a08ef36925e88790b328afe49295486f79e706e25" exitCode=0
Sep 08 16:58:15.513657 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:15.513530    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d" event={"ID":"135ef8c3-562b-4dbf-8864-36b4152c69d7","Type":"ContainerDied","Data":"5f2a7a8fac8f88a25775d72a08ef36925e88790b328afe49295486f79e706e25"}
Sep 08 16:58:15.644474 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:15.644446    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:58:15.711710 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:15.711671    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/135ef8c3-562b-4dbf-8864-36b4152c69d7-home\") pod \"135ef8c3-562b-4dbf-8864-36b4152c69d7\" (UID: \"135ef8c3-562b-4dbf-8864-36b4152c69d7\") "
Sep 08 16:58:15.711916 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:15.711721    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tlnm\" (UniqueName: \"kubernetes.io/projected/135ef8c3-562b-4dbf-8864-36b4152c69d7-kube-api-access-2tlnm\") pod \"135ef8c3-562b-4dbf-8864-36b4152c69d7\" (UID: \"135ef8c3-562b-4dbf-8864-36b4152c69d7\") "
Sep 08 16:58:15.711916 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:15.711758    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/135ef8c3-562b-4dbf-8864-36b4152c69d7-dshm\") pod \"135ef8c3-562b-4dbf-8864-36b4152c69d7\" (UID: \"135ef8c3-562b-4dbf-8864-36b4152c69d7\") "
Sep 08 16:58:15.711916 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:15.711827    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/135ef8c3-562b-4dbf-8864-36b4152c69d7-model-cache\") pod \"135ef8c3-562b-4dbf-8864-36b4152c69d7\" (UID: \"135ef8c3-562b-4dbf-8864-36b4152c69d7\") "
Sep 08 16:58:15.712091 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:15.711932    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/135ef8c3-562b-4dbf-8864-36b4152c69d7-tls-certs\") pod \"135ef8c3-562b-4dbf-8864-36b4152c69d7\" (UID: \"135ef8c3-562b-4dbf-8864-36b4152c69d7\") "
Sep 08 16:58:15.712091 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:15.711975    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/135ef8c3-562b-4dbf-8864-36b4152c69d7-tmp-dir\") pod \"135ef8c3-562b-4dbf-8864-36b4152c69d7\" (UID: \"135ef8c3-562b-4dbf-8864-36b4152c69d7\") "
Sep 08 16:58:15.712091 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:15.711975    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135ef8c3-562b-4dbf-8864-36b4152c69d7-home" (OuterVolumeSpecName: "home") pod "135ef8c3-562b-4dbf-8864-36b4152c69d7" (UID: "135ef8c3-562b-4dbf-8864-36b4152c69d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:58:15.712263 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:15.712078    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135ef8c3-562b-4dbf-8864-36b4152c69d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "135ef8c3-562b-4dbf-8864-36b4152c69d7" (UID: "135ef8c3-562b-4dbf-8864-36b4152c69d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:58:15.712377 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:15.712285    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/135ef8c3-562b-4dbf-8864-36b4152c69d7-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:58:15.712377 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:15.712351    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/135ef8c3-562b-4dbf-8864-36b4152c69d7-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:58:15.712377 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:15.712355    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135ef8c3-562b-4dbf-8864-36b4152c69d7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "135ef8c3-562b-4dbf-8864-36b4152c69d7" (UID: "135ef8c3-562b-4dbf-8864-36b4152c69d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:58:15.714272 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:15.714243    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135ef8c3-562b-4dbf-8864-36b4152c69d7-dshm" (OuterVolumeSpecName: "dshm") pod "135ef8c3-562b-4dbf-8864-36b4152c69d7" (UID: "135ef8c3-562b-4dbf-8864-36b4152c69d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:58:15.714272 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:15.714249    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135ef8c3-562b-4dbf-8864-36b4152c69d7-kube-api-access-2tlnm" (OuterVolumeSpecName: "kube-api-access-2tlnm") pod "135ef8c3-562b-4dbf-8864-36b4152c69d7" (UID: "135ef8c3-562b-4dbf-8864-36b4152c69d7"). InnerVolumeSpecName "kube-api-access-2tlnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:58:15.714407 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:15.714275    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/135ef8c3-562b-4dbf-8864-36b4152c69d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "135ef8c3-562b-4dbf-8864-36b4152c69d7" (UID: "135ef8c3-562b-4dbf-8864-36b4152c69d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:58:15.813722 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:15.813677    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/135ef8c3-562b-4dbf-8864-36b4152c69d7-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:58:15.813722 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:15.813714    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/135ef8c3-562b-4dbf-8864-36b4152c69d7-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:58:15.813722 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:15.813723    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tlnm\" (UniqueName: \"kubernetes.io/projected/135ef8c3-562b-4dbf-8864-36b4152c69d7-kube-api-access-2tlnm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:58:15.813722 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:15.813733    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/135ef8c3-562b-4dbf-8864-36b4152c69d7-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:58:16.520193 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:16.520162    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d" event={"ID":"135ef8c3-562b-4dbf-8864-36b4152c69d7","Type":"ContainerDied","Data":"b78003245f45b72bcecabca8bb576f0d2675d216c3915e5177482b02e4711f29"}
Sep 08 16:58:16.520193 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:16.520195    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"
Sep 08 16:58:16.520696 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:16.520211    2557 scope.go:117] "RemoveContainer" containerID="5f2a7a8fac8f88a25775d72a08ef36925e88790b328afe49295486f79e706e25"
Sep 08 16:58:16.541981 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:16.541948    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"]
Sep 08 16:58:16.552684 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:16.552658    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-d2b1cc/ipp-route-d2b1cc-kserve-658db547f9-c5x4d"]
Sep 08 16:58:18.345410 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:58:18.345378    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="135ef8c3-562b-4dbf-8864-36b4152c69d7" path="/var/lib/kubelet/pods/135ef8c3-562b-4dbf-8864-36b4152c69d7/volumes"
Sep 08 16:59:09.050126 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:09.050090    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-84ba67-6hbhr"]
Sep 08 16:59:09.050650 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:09.050474    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5b59a4e-8704-4a98-8d12-a4afe8d1e126" containerName="revoke-keys"
Sep 08 16:59:09.050650 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:09.050485    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5b59a4e-8704-4a98-8d12-a4afe8d1e126" containerName="revoke-keys"
Sep 08 16:59:09.050650 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:09.050499    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2189b15-dc78-4597-a0ff-1bec9bf395ad" containerName="payload-processing"
Sep 08 16:59:09.050650 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:09.050505    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2189b15-dc78-4597-a0ff-1bec9bf395ad" containerName="payload-processing"
Sep 08 16:59:09.050650 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:09.050513    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="135ef8c3-562b-4dbf-8864-36b4152c69d7" containerName="main"
Sep 08 16:59:09.050650 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:09.050518    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="135ef8c3-562b-4dbf-8864-36b4152c69d7" containerName="main"
Sep 08 16:59:09.050650 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:09.050570    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="135ef8c3-562b-4dbf-8864-36b4152c69d7" containerName="main"
Sep 08 16:59:09.050650 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:09.050578    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2189b15-dc78-4597-a0ff-1bec9bf395ad" containerName="payload-processing"
Sep 08 16:59:09.050650 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:09.050585    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5b59a4e-8704-4a98-8d12-a4afe8d1e126" containerName="revoke-keys"
Sep 08 16:59:09.053948 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:09.053929    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-84ba67-6hbhr"
Sep 08 16:59:09.056911 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:09.056881    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:59:09.057073 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:09.056887    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-886jf\""
Sep 08 16:59:09.063020 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:09.062988    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-84ba67-6hbhr"]
Sep 08 16:59:09.072201 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:09.072164    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/176b67a7-41af-4f2c-bcb4-2fea3bcba751-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-84ba67-6hbhr\" (UID: \"176b67a7-41af-4f2c-bcb4-2fea3bcba751\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-84ba67-6hbhr"
Sep 08 16:59:09.173147 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:09.173082    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/176b67a7-41af-4f2c-bcb4-2fea3bcba751-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-84ba67-6hbhr\" (UID: \"176b67a7-41af-4f2c-bcb4-2fea3bcba751\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-84ba67-6hbhr"
Sep 08 16:59:09.174104 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:09.174076    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/176b67a7-41af-4f2c-bcb4-2fea3bcba751-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-84ba67-6hbhr\" (UID: \"176b67a7-41af-4f2c-bcb4-2fea3bcba751\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-84ba67-6hbhr"
Sep 08 16:59:09.365116 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:09.365025    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-84ba67-6hbhr"
Sep 08 16:59:09.659802 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:59:09.659769    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176b67a7_41af_4f2c_bcb4_2fea3bcba751.slice/crio-cdd3c06bf5b60823582bb696d0399f16c79370aa193cb824ad0a94cc19332c55 WatchSource:0}: Error finding container cdd3c06bf5b60823582bb696d0399f16c79370aa193cb824ad0a94cc19332c55: Status 404 returned error can't find the container with id cdd3c06bf5b60823582bb696d0399f16c79370aa193cb824ad0a94cc19332c55
Sep 08 16:59:09.661389 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:09.661365    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-84ba67-6hbhr"]
Sep 08 16:59:09.786229 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:09.786189    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-84ba67-6hbhr" event={"ID":"176b67a7-41af-4f2c-bcb4-2fea3bcba751","Type":"ContainerStarted","Data":"cdd3c06bf5b60823582bb696d0399f16c79370aa193cb824ad0a94cc19332c55"}
Sep 08 16:59:10.792688 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:10.792654    2557 generic.go:356] "Generic (PLEG): container finished" podID="176b67a7-41af-4f2c-bcb4-2fea3bcba751" containerID="a6698c86034fd643c87f7811a26f90fd87193524e990e948e943237a00a8b035" exitCode=0
Sep 08 16:59:10.792688 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:10.792694    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-84ba67-6hbhr" event={"ID":"176b67a7-41af-4f2c-bcb4-2fea3bcba751","Type":"ContainerDied","Data":"a6698c86034fd643c87f7811a26f90fd87193524e990e948e943237a00a8b035"}
Sep 08 16:59:11.937247 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:11.937222    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-84ba67-6hbhr"
Sep 08 16:59:11.997154 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:11.997123    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/176b67a7-41af-4f2c-bcb4-2fea3bcba751-maas-api-service-ca\") pod \"176b67a7-41af-4f2c-bcb4-2fea3bcba751\" (UID: \"176b67a7-41af-4f2c-bcb4-2fea3bcba751\") "
Sep 08 16:59:11.997523 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:11.997494    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/176b67a7-41af-4f2c-bcb4-2fea3bcba751-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "176b67a7-41af-4f2c-bcb4-2fea3bcba751" (UID: "176b67a7-41af-4f2c-bcb4-2fea3bcba751"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:59:12.098479 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:12.098378    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/176b67a7-41af-4f2c-bcb4-2fea3bcba751-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:59:12.804269 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:12.804236    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-84ba67-6hbhr" event={"ID":"176b67a7-41af-4f2c-bcb4-2fea3bcba751","Type":"ContainerDied","Data":"cdd3c06bf5b60823582bb696d0399f16c79370aa193cb824ad0a94cc19332c55"}
Sep 08 16:59:12.804269 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:12.804268    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-84ba67-6hbhr"
Sep 08 16:59:12.804483 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:12.804284    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdd3c06bf5b60823582bb696d0399f16c79370aa193cb824ad0a94cc19332c55"
Sep 08 16:59:19.896793 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:19.896754    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r"]
Sep 08 16:59:19.897274 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:19.896989    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r" podUID="f5bb403a-fb66-4d6a-ae69-57ee740b2d56" containerName="payload-processing" containerID="cri-o://96664e909c91b1c3064e3431aef640a01992334e01724bf7243930d167f85cd3" gracePeriod=30
Sep 08 16:59:19.907339 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:19.907262    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l"]
Sep 08 16:59:19.908087 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:19.908045    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l" podUID="c7559184-0c46-4686-83fb-1272e0a22579" containerName="payload-pre-processing" containerID="cri-o://8f0e2308729a185fa84d81989fd2192bfaec4d35003ce3af4aa6ff04984bcf42" gracePeriod=30
Sep 08 16:59:20.098377 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.098341    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-84ba67-6hbhr"]
Sep 08 16:59:20.102981 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.102946    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-84ba67-6hbhr"]
Sep 08 16:59:20.161628 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.161606    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r"
Sep 08 16:59:20.201030 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.200997    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l"
Sep 08 16:59:20.264872 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.264819    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvttn\" (UniqueName: \"kubernetes.io/projected/c7559184-0c46-4686-83fb-1272e0a22579-kube-api-access-mvttn\") pod \"c7559184-0c46-4686-83fb-1272e0a22579\" (UID: \"c7559184-0c46-4686-83fb-1272e0a22579\") "
Sep 08 16:59:20.265082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.264907    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znvt4\" (UniqueName: \"kubernetes.io/projected/f5bb403a-fb66-4d6a-ae69-57ee740b2d56-kube-api-access-znvt4\") pod \"f5bb403a-fb66-4d6a-ae69-57ee740b2d56\" (UID: \"f5bb403a-fb66-4d6a-ae69-57ee740b2d56\") "
Sep 08 16:59:20.265082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.264980    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bb403a-fb66-4d6a-ae69-57ee740b2d56-plugins-config-volume\") pod \"f5bb403a-fb66-4d6a-ae69-57ee740b2d56\" (UID: \"f5bb403a-fb66-4d6a-ae69-57ee740b2d56\") "
Sep 08 16:59:20.265082 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.264997    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7559184-0c46-4686-83fb-1272e0a22579-plugins-config-volume\") pod \"c7559184-0c46-4686-83fb-1272e0a22579\" (UID: \"c7559184-0c46-4686-83fb-1272e0a22579\") "
Sep 08 16:59:20.265430 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.265404    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5bb403a-fb66-4d6a-ae69-57ee740b2d56-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f5bb403a-fb66-4d6a-ae69-57ee740b2d56" (UID: "f5bb403a-fb66-4d6a-ae69-57ee740b2d56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:59:20.265497 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.265404    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7559184-0c46-4686-83fb-1272e0a22579-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c7559184-0c46-4686-83fb-1272e0a22579" (UID: "c7559184-0c46-4686-83fb-1272e0a22579"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:59:20.267168 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.267144    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5bb403a-fb66-4d6a-ae69-57ee740b2d56-kube-api-access-znvt4" (OuterVolumeSpecName: "kube-api-access-znvt4") pod "f5bb403a-fb66-4d6a-ae69-57ee740b2d56" (UID: "f5bb403a-fb66-4d6a-ae69-57ee740b2d56"). InnerVolumeSpecName "kube-api-access-znvt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:59:20.267245 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.267148    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7559184-0c46-4686-83fb-1272e0a22579-kube-api-access-mvttn" (OuterVolumeSpecName: "kube-api-access-mvttn") pod "c7559184-0c46-4686-83fb-1272e0a22579" (UID: "c7559184-0c46-4686-83fb-1272e0a22579"). InnerVolumeSpecName "kube-api-access-mvttn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:59:20.346874 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.346818    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="176b67a7-41af-4f2c-bcb4-2fea3bcba751" path="/var/lib/kubelet/pods/176b67a7-41af-4f2c-bcb4-2fea3bcba751/volumes"
Sep 08 16:59:20.366244 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.366194    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znvt4\" (UniqueName: \"kubernetes.io/projected/f5bb403a-fb66-4d6a-ae69-57ee740b2d56-kube-api-access-znvt4\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:59:20.366244 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.366240    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bb403a-fb66-4d6a-ae69-57ee740b2d56-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:59:20.366244 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.366252    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7559184-0c46-4686-83fb-1272e0a22579-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:59:20.366244 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.366262    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvttn\" (UniqueName: \"kubernetes.io/projected/c7559184-0c46-4686-83fb-1272e0a22579-kube-api-access-mvttn\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:59:20.846108 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.846069    2557 generic.go:356] "Generic (PLEG): container finished" podID="f5bb403a-fb66-4d6a-ae69-57ee740b2d56" containerID="96664e909c91b1c3064e3431aef640a01992334e01724bf7243930d167f85cd3" exitCode=0
Sep 08 16:59:20.846328 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.846137    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r"
Sep 08 16:59:20.846328 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.846157    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r" event={"ID":"f5bb403a-fb66-4d6a-ae69-57ee740b2d56","Type":"ContainerDied","Data":"96664e909c91b1c3064e3431aef640a01992334e01724bf7243930d167f85cd3"}
Sep 08 16:59:20.846328 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.846255    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r" event={"ID":"f5bb403a-fb66-4d6a-ae69-57ee740b2d56","Type":"ContainerDied","Data":"c0d351dc3ac401146e955747b97b6c5e8a9ddb8851388a38128ad3de7a7e6390"}
Sep 08 16:59:20.846328 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.846283    2557 scope.go:117] "RemoveContainer" containerID="96664e909c91b1c3064e3431aef640a01992334e01724bf7243930d167f85cd3"
Sep 08 16:59:20.847676 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.847649    2557 generic.go:356] "Generic (PLEG): container finished" podID="c7559184-0c46-4686-83fb-1272e0a22579" containerID="8f0e2308729a185fa84d81989fd2192bfaec4d35003ce3af4aa6ff04984bcf42" exitCode=0
Sep 08 16:59:20.847806 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.847706    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l" event={"ID":"c7559184-0c46-4686-83fb-1272e0a22579","Type":"ContainerDied","Data":"8f0e2308729a185fa84d81989fd2192bfaec4d35003ce3af4aa6ff04984bcf42"}
Sep 08 16:59:20.847806 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.847725    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l"
Sep 08 16:59:20.847806 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.847754    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l" event={"ID":"c7559184-0c46-4686-83fb-1272e0a22579","Type":"ContainerDied","Data":"1aadb3ce8d673f0a1ff4de8f68428fd1452b277a3cbe75b708dbc66dd9b7132c"}
Sep 08 16:59:20.856259 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.856235    2557 scope.go:117] "RemoveContainer" containerID="96664e909c91b1c3064e3431aef640a01992334e01724bf7243930d167f85cd3"
Sep 08 16:59:20.856624 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:59:20.856604    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96664e909c91b1c3064e3431aef640a01992334e01724bf7243930d167f85cd3\": container with ID starting with 96664e909c91b1c3064e3431aef640a01992334e01724bf7243930d167f85cd3 not found: ID does not exist" containerID="96664e909c91b1c3064e3431aef640a01992334e01724bf7243930d167f85cd3"
Sep 08 16:59:20.856664 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.856635    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96664e909c91b1c3064e3431aef640a01992334e01724bf7243930d167f85cd3"} err="failed to get container status \"96664e909c91b1c3064e3431aef640a01992334e01724bf7243930d167f85cd3\": rpc error: code = NotFound desc = could not find container \"96664e909c91b1c3064e3431aef640a01992334e01724bf7243930d167f85cd3\": container with ID starting with 96664e909c91b1c3064e3431aef640a01992334e01724bf7243930d167f85cd3 not found: ID does not exist"
Sep 08 16:59:20.856664 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.856652    2557 scope.go:117] "RemoveContainer" containerID="8f0e2308729a185fa84d81989fd2192bfaec4d35003ce3af4aa6ff04984bcf42"
Sep 08 16:59:20.865299 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.865270    2557 scope.go:117] "RemoveContainer" containerID="8f0e2308729a185fa84d81989fd2192bfaec4d35003ce3af4aa6ff04984bcf42"
Sep 08 16:59:20.865654 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:59:20.865634    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0e2308729a185fa84d81989fd2192bfaec4d35003ce3af4aa6ff04984bcf42\": container with ID starting with 8f0e2308729a185fa84d81989fd2192bfaec4d35003ce3af4aa6ff04984bcf42 not found: ID does not exist" containerID="8f0e2308729a185fa84d81989fd2192bfaec4d35003ce3af4aa6ff04984bcf42"
Sep 08 16:59:20.865717 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.865665    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0e2308729a185fa84d81989fd2192bfaec4d35003ce3af4aa6ff04984bcf42"} err="failed to get container status \"8f0e2308729a185fa84d81989fd2192bfaec4d35003ce3af4aa6ff04984bcf42\": rpc error: code = NotFound desc = could not find container \"8f0e2308729a185fa84d81989fd2192bfaec4d35003ce3af4aa6ff04984bcf42\": container with ID starting with 8f0e2308729a185fa84d81989fd2192bfaec4d35003ce3af4aa6ff04984bcf42 not found: ID does not exist"
Sep 08 16:59:20.868283 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.868252    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r"]
Sep 08 16:59:20.875682 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.875645    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-84ba67-767f85d4f5-4h89r"]
Sep 08 16:59:20.887234 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.887185    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l"]
Sep 08 16:59:20.891084 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:20.891053    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-84ba67-7bd7f889db-nsx7l"]
Sep 08 16:59:22.346076 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:22.346038    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7559184-0c46-4686-83fb-1272e0a22579" path="/var/lib/kubelet/pods/c7559184-0c46-4686-83fb-1272e0a22579/volumes"
Sep 08 16:59:22.346465 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:22.346376    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5bb403a-fb66-4d6a-ae69-57ee740b2d56" path="/var/lib/kubelet/pods/f5bb403a-fb66-4d6a-ae69-57ee740b2d56/volumes"
Sep 08 16:59:23.237758 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:23.237726    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:59:27.428537 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:27.428500    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-608268-vgwxf"]
Sep 08 16:59:27.428963 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:27.428891    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="176b67a7-41af-4f2c-bcb4-2fea3bcba751" containerName="revoke-keys"
Sep 08 16:59:27.428963 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:27.428906    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="176b67a7-41af-4f2c-bcb4-2fea3bcba751" containerName="revoke-keys"
Sep 08 16:59:27.428963 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:27.428916    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7559184-0c46-4686-83fb-1272e0a22579" containerName="payload-pre-processing"
Sep 08 16:59:27.428963 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:27.428921    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7559184-0c46-4686-83fb-1272e0a22579" containerName="payload-pre-processing"
Sep 08 16:59:27.428963 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:27.428935    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5bb403a-fb66-4d6a-ae69-57ee740b2d56" containerName="payload-processing"
Sep 08 16:59:27.428963 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:27.428941    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5bb403a-fb66-4d6a-ae69-57ee740b2d56" containerName="payload-processing"
Sep 08 16:59:27.429154 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:27.429000    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="176b67a7-41af-4f2c-bcb4-2fea3bcba751" containerName="revoke-keys"
Sep 08 16:59:27.429154 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:27.429011    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5bb403a-fb66-4d6a-ae69-57ee740b2d56" containerName="payload-processing"
Sep 08 16:59:27.429154 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:27.429019    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7559184-0c46-4686-83fb-1272e0a22579" containerName="payload-pre-processing"
Sep 08 16:59:27.433442 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:27.433422    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-608268-vgwxf"
Sep 08 16:59:27.437030 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:27.437004    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:59:27.437160 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:27.437119    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-886jf\""
Sep 08 16:59:27.451441 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:27.451409    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-608268-vgwxf"]
Sep 08 16:59:27.534826 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:27.534787    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c9fcce-dff4-4448-8c21-090e17984c1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-608268-vgwxf\" (UID: \"b3c9fcce-dff4-4448-8c21-090e17984c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-608268-vgwxf"
Sep 08 16:59:27.635593 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:27.635521    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c9fcce-dff4-4448-8c21-090e17984c1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-608268-vgwxf\" (UID: \"b3c9fcce-dff4-4448-8c21-090e17984c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-608268-vgwxf"
Sep 08 16:59:27.636418 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:27.636390    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c9fcce-dff4-4448-8c21-090e17984c1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-608268-vgwxf\" (UID: \"b3c9fcce-dff4-4448-8c21-090e17984c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-608268-vgwxf"
Sep 08 16:59:27.744331 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:27.744233    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-608268-vgwxf"
Sep 08 16:59:28.017076 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:28.017048    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-608268-vgwxf"]
Sep 08 16:59:28.019314 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:59:28.019288    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c9fcce_dff4_4448_8c21_090e17984c1c.slice/crio-993d4834d84c96b3572728b0122afe09077b09a10f1d90b74cc936710082f80c WatchSource:0}: Error finding container 993d4834d84c96b3572728b0122afe09077b09a10f1d90b74cc936710082f80c: Status 404 returned error can't find the container with id 993d4834d84c96b3572728b0122afe09077b09a10f1d90b74cc936710082f80c
Sep 08 16:59:28.893919 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:28.893872    2557 generic.go:356] "Generic (PLEG): container finished" podID="b3c9fcce-dff4-4448-8c21-090e17984c1c" containerID="4fcc7f636a02e01d2971fac6051f0a3467f253ba401adc8fa2bf2f0e42e26b3d" exitCode=0
Sep 08 16:59:28.894314 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:28.893963    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-608268-vgwxf" event={"ID":"b3c9fcce-dff4-4448-8c21-090e17984c1c","Type":"ContainerDied","Data":"4fcc7f636a02e01d2971fac6051f0a3467f253ba401adc8fa2bf2f0e42e26b3d"}
Sep 08 16:59:28.894314 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:28.894012    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-608268-vgwxf" event={"ID":"b3c9fcce-dff4-4448-8c21-090e17984c1c","Type":"ContainerStarted","Data":"993d4834d84c96b3572728b0122afe09077b09a10f1d90b74cc936710082f80c"}
Sep 08 16:59:30.034428 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:30.034404    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-608268-vgwxf"
Sep 08 16:59:30.155889 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:30.155776    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c9fcce-dff4-4448-8c21-090e17984c1c-maas-api-service-ca\") pod \"b3c9fcce-dff4-4448-8c21-090e17984c1c\" (UID: \"b3c9fcce-dff4-4448-8c21-090e17984c1c\") "
Sep 08 16:59:30.156167 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:30.156140    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3c9fcce-dff4-4448-8c21-090e17984c1c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b3c9fcce-dff4-4448-8c21-090e17984c1c" (UID: "b3c9fcce-dff4-4448-8c21-090e17984c1c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:59:30.256578 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:30.256519    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c9fcce-dff4-4448-8c21-090e17984c1c-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:59:30.905371 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:30.905333    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-608268-vgwxf" event={"ID":"b3c9fcce-dff4-4448-8c21-090e17984c1c","Type":"ContainerDied","Data":"993d4834d84c96b3572728b0122afe09077b09a10f1d90b74cc936710082f80c"}
Sep 08 16:59:30.905551 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:30.905386    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="993d4834d84c96b3572728b0122afe09077b09a10f1d90b74cc936710082f80c"
Sep 08 16:59:30.905551 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:30.905348    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-608268-vgwxf"
Sep 08 16:59:37.975242 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:37.975187    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh"]
Sep 08 16:59:37.975747 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:37.975429    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh" podUID="4c90ce47-9dbe-42f4-b9a9-a448567a00d7" containerName="payload-processing" containerID="cri-o://363025dfcd6c2e2eaa8785969e8d78f9ed3ffd4be20220f7158546f7cf750185" gracePeriod=30
Sep 08 16:59:37.987635 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:37.987601    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg"]
Sep 08 16:59:37.988452 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:37.988028    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg" podUID="4e0f270f-0dcb-4261-baea-8a77e154f53b" containerName="payload-pre-processing" containerID="cri-o://1306b5c9dd1be4edb9d3289831eb80dac143fdb654ec3d6d83ecd533c0875d61" gracePeriod=30
Sep 08 16:59:38.173083 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.173044    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-608268-vgwxf"]
Sep 08 16:59:38.177303 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.177269    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-608268-vgwxf"]
Sep 08 16:59:38.244861 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.244821    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh"
Sep 08 16:59:38.285024 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.284998    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg"
Sep 08 16:59:38.325789 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.325755    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdkxk\" (UniqueName: \"kubernetes.io/projected/4c90ce47-9dbe-42f4-b9a9-a448567a00d7-kube-api-access-wdkxk\") pod \"4c90ce47-9dbe-42f4-b9a9-a448567a00d7\" (UID: \"4c90ce47-9dbe-42f4-b9a9-a448567a00d7\") "
Sep 08 16:59:38.326000 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.325844    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c90ce47-9dbe-42f4-b9a9-a448567a00d7-plugins-config-volume\") pod \"4c90ce47-9dbe-42f4-b9a9-a448567a00d7\" (UID: \"4c90ce47-9dbe-42f4-b9a9-a448567a00d7\") "
Sep 08 16:59:38.326232 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.326210    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c90ce47-9dbe-42f4-b9a9-a448567a00d7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4c90ce47-9dbe-42f4-b9a9-a448567a00d7" (UID: "4c90ce47-9dbe-42f4-b9a9-a448567a00d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:59:38.328339 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.328308    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c90ce47-9dbe-42f4-b9a9-a448567a00d7-kube-api-access-wdkxk" (OuterVolumeSpecName: "kube-api-access-wdkxk") pod "4c90ce47-9dbe-42f4-b9a9-a448567a00d7" (UID: "4c90ce47-9dbe-42f4-b9a9-a448567a00d7"). InnerVolumeSpecName "kube-api-access-wdkxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:59:38.345442 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.345401    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3c9fcce-dff4-4448-8c21-090e17984c1c" path="/var/lib/kubelet/pods/b3c9fcce-dff4-4448-8c21-090e17984c1c/volumes"
Sep 08 16:59:38.426522 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.426480    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rh8q\" (UniqueName: \"kubernetes.io/projected/4e0f270f-0dcb-4261-baea-8a77e154f53b-kube-api-access-7rh8q\") pod \"4e0f270f-0dcb-4261-baea-8a77e154f53b\" (UID: \"4e0f270f-0dcb-4261-baea-8a77e154f53b\") "
Sep 08 16:59:38.426707 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.426595    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e0f270f-0dcb-4261-baea-8a77e154f53b-plugins-config-volume\") pod \"4e0f270f-0dcb-4261-baea-8a77e154f53b\" (UID: \"4e0f270f-0dcb-4261-baea-8a77e154f53b\") "
Sep 08 16:59:38.426795 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.426725    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c90ce47-9dbe-42f4-b9a9-a448567a00d7-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:59:38.426795 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.426736    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdkxk\" (UniqueName: \"kubernetes.io/projected/4c90ce47-9dbe-42f4-b9a9-a448567a00d7-kube-api-access-wdkxk\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:59:38.427058 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.427031    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e0f270f-0dcb-4261-baea-8a77e154f53b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4e0f270f-0dcb-4261-baea-8a77e154f53b" (UID: "4e0f270f-0dcb-4261-baea-8a77e154f53b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:59:38.428742 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.428718    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e0f270f-0dcb-4261-baea-8a77e154f53b-kube-api-access-7rh8q" (OuterVolumeSpecName: "kube-api-access-7rh8q") pod "4e0f270f-0dcb-4261-baea-8a77e154f53b" (UID: "4e0f270f-0dcb-4261-baea-8a77e154f53b"). InnerVolumeSpecName "kube-api-access-7rh8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:59:38.527797 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.527701    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e0f270f-0dcb-4261-baea-8a77e154f53b-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:59:38.527797 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.527737    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rh8q\" (UniqueName: \"kubernetes.io/projected/4e0f270f-0dcb-4261-baea-8a77e154f53b-kube-api-access-7rh8q\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:59:38.949844 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.949804    2557 generic.go:356] "Generic (PLEG): container finished" podID="4c90ce47-9dbe-42f4-b9a9-a448567a00d7" containerID="363025dfcd6c2e2eaa8785969e8d78f9ed3ffd4be20220f7158546f7cf750185" exitCode=0
Sep 08 16:59:38.950062 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.949869    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh" event={"ID":"4c90ce47-9dbe-42f4-b9a9-a448567a00d7","Type":"ContainerDied","Data":"363025dfcd6c2e2eaa8785969e8d78f9ed3ffd4be20220f7158546f7cf750185"}
Sep 08 16:59:38.950062 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.949883    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh"
Sep 08 16:59:38.950062 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.949974    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh" event={"ID":"4c90ce47-9dbe-42f4-b9a9-a448567a00d7","Type":"ContainerDied","Data":"a2efbb7c54f01e90c9e5cf135e116fd691e77562e9361d5006ec91a96c0370f6"}
Sep 08 16:59:38.950062 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.950007    2557 scope.go:117] "RemoveContainer" containerID="363025dfcd6c2e2eaa8785969e8d78f9ed3ffd4be20220f7158546f7cf750185"
Sep 08 16:59:38.951564 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.951522    2557 generic.go:356] "Generic (PLEG): container finished" podID="4e0f270f-0dcb-4261-baea-8a77e154f53b" containerID="1306b5c9dd1be4edb9d3289831eb80dac143fdb654ec3d6d83ecd533c0875d61" exitCode=0
Sep 08 16:59:38.951564 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.951554    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg" event={"ID":"4e0f270f-0dcb-4261-baea-8a77e154f53b","Type":"ContainerDied","Data":"1306b5c9dd1be4edb9d3289831eb80dac143fdb654ec3d6d83ecd533c0875d61"}
Sep 08 16:59:38.951712 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.951602    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg"
Sep 08 16:59:38.951712 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.951611    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg" event={"ID":"4e0f270f-0dcb-4261-baea-8a77e154f53b","Type":"ContainerDied","Data":"3c4ee9dcbb5bb38e6b5b09e21dded109f37a490b54421a4b03c7cc01cac6fcd5"}
Sep 08 16:59:38.959452 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.959356    2557 scope.go:117] "RemoveContainer" containerID="363025dfcd6c2e2eaa8785969e8d78f9ed3ffd4be20220f7158546f7cf750185"
Sep 08 16:59:38.959740 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:59:38.959713    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"363025dfcd6c2e2eaa8785969e8d78f9ed3ffd4be20220f7158546f7cf750185\": container with ID starting with 363025dfcd6c2e2eaa8785969e8d78f9ed3ffd4be20220f7158546f7cf750185 not found: ID does not exist" containerID="363025dfcd6c2e2eaa8785969e8d78f9ed3ffd4be20220f7158546f7cf750185"
Sep 08 16:59:38.959808 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.959759    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"363025dfcd6c2e2eaa8785969e8d78f9ed3ffd4be20220f7158546f7cf750185"} err="failed to get container status \"363025dfcd6c2e2eaa8785969e8d78f9ed3ffd4be20220f7158546f7cf750185\": rpc error: code = NotFound desc = could not find container \"363025dfcd6c2e2eaa8785969e8d78f9ed3ffd4be20220f7158546f7cf750185\": container with ID starting with 363025dfcd6c2e2eaa8785969e8d78f9ed3ffd4be20220f7158546f7cf750185 not found: ID does not exist"
Sep 08 16:59:38.959808 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.959781    2557 scope.go:117] "RemoveContainer" containerID="1306b5c9dd1be4edb9d3289831eb80dac143fdb654ec3d6d83ecd533c0875d61"
Sep 08 16:59:38.968065 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.968027    2557 scope.go:117] "RemoveContainer" containerID="1306b5c9dd1be4edb9d3289831eb80dac143fdb654ec3d6d83ecd533c0875d61"
Sep 08 16:59:38.968463 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:59:38.968440    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1306b5c9dd1be4edb9d3289831eb80dac143fdb654ec3d6d83ecd533c0875d61\": container with ID starting with 1306b5c9dd1be4edb9d3289831eb80dac143fdb654ec3d6d83ecd533c0875d61 not found: ID does not exist" containerID="1306b5c9dd1be4edb9d3289831eb80dac143fdb654ec3d6d83ecd533c0875d61"
Sep 08 16:59:38.968541 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.968471    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1306b5c9dd1be4edb9d3289831eb80dac143fdb654ec3d6d83ecd533c0875d61"} err="failed to get container status \"1306b5c9dd1be4edb9d3289831eb80dac143fdb654ec3d6d83ecd533c0875d61\": rpc error: code = NotFound desc = could not find container \"1306b5c9dd1be4edb9d3289831eb80dac143fdb654ec3d6d83ecd533c0875d61\": container with ID starting with 1306b5c9dd1be4edb9d3289831eb80dac143fdb654ec3d6d83ecd533c0875d61 not found: ID does not exist"
Sep 08 16:59:38.970383 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.970363    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh"]
Sep 08 16:59:38.975243 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.975222    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-608268-7b54d4c49f-2tndh"]
Sep 08 16:59:38.985486 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.985459    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg"]
Sep 08 16:59:38.987880 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:38.987853    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-608268-54ff64d8bb-zkdrg"]
Sep 08 16:59:40.345539 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:40.345503    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c90ce47-9dbe-42f4-b9a9-a448567a00d7" path="/var/lib/kubelet/pods/4c90ce47-9dbe-42f4-b9a9-a448567a00d7/volumes"
Sep 08 16:59:40.346031 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:40.345856    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e0f270f-0dcb-4261-baea-8a77e154f53b" path="/var/lib/kubelet/pods/4e0f270f-0dcb-4261-baea-8a77e154f53b/volumes"
Sep 08 16:59:40.849922 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:40.849890    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 16:59:45.542610 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:45.542573    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b15ca-h4dwk"]
Sep 08 16:59:45.543032 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:45.542948    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e0f270f-0dcb-4261-baea-8a77e154f53b" containerName="payload-pre-processing"
Sep 08 16:59:45.543032 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:45.542960    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e0f270f-0dcb-4261-baea-8a77e154f53b" containerName="payload-pre-processing"
Sep 08 16:59:45.543032 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:45.542968    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c90ce47-9dbe-42f4-b9a9-a448567a00d7" containerName="payload-processing"
Sep 08 16:59:45.543032 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:45.542974    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c90ce47-9dbe-42f4-b9a9-a448567a00d7" containerName="payload-processing"
Sep 08 16:59:45.543032 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:45.542992    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3c9fcce-dff4-4448-8c21-090e17984c1c" containerName="revoke-keys"
Sep 08 16:59:45.543032 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:45.542999    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3c9fcce-dff4-4448-8c21-090e17984c1c" containerName="revoke-keys"
Sep 08 16:59:45.543234 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:45.543060    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3c9fcce-dff4-4448-8c21-090e17984c1c" containerName="revoke-keys"
Sep 08 16:59:45.543234 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:45.543069    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c90ce47-9dbe-42f4-b9a9-a448567a00d7" containerName="payload-processing"
Sep 08 16:59:45.543234 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:45.543078    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e0f270f-0dcb-4261-baea-8a77e154f53b" containerName="payload-pre-processing"
Sep 08 16:59:45.547390 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:45.547367    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b15ca-h4dwk"
Sep 08 16:59:45.550271 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:45.550244    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:59:45.550559 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:45.550543    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-886jf\""
Sep 08 16:59:45.554388 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:45.554342    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b15ca-h4dwk"]
Sep 08 16:59:45.687510 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:45.687469    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00a8a3b-f3da-443c-aaa1-99dc90c2a3d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-6b15ca-h4dwk\" (UID: \"a00a8a3b-f3da-443c-aaa1-99dc90c2a3d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b15ca-h4dwk"
Sep 08 16:59:45.788760 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:45.788717    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00a8a3b-f3da-443c-aaa1-99dc90c2a3d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-6b15ca-h4dwk\" (UID: \"a00a8a3b-f3da-443c-aaa1-99dc90c2a3d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b15ca-h4dwk"
Sep 08 16:59:45.789453 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:45.789428    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00a8a3b-f3da-443c-aaa1-99dc90c2a3d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-6b15ca-h4dwk\" (UID: \"a00a8a3b-f3da-443c-aaa1-99dc90c2a3d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b15ca-h4dwk"
Sep 08 16:59:45.859537 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:45.859437    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b15ca-h4dwk"
Sep 08 16:59:46.132139 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:46.132114    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b15ca-h4dwk"]
Sep 08 16:59:46.134992 ip-10-0-139-51 kubenswrapper[2557]: W0908 16:59:46.134962    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00a8a3b_f3da_443c_aaa1_99dc90c2a3d3.slice/crio-8200bfa1017fd07314a5790581bb4819c92b1ce2d1b8d9af7468023e3c2aee6d WatchSource:0}: Error finding container 8200bfa1017fd07314a5790581bb4819c92b1ce2d1b8d9af7468023e3c2aee6d: Status 404 returned error can't find the container with id 8200bfa1017fd07314a5790581bb4819c92b1ce2d1b8d9af7468023e3c2aee6d
Sep 08 16:59:46.998403 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:46.998369    2557 generic.go:356] "Generic (PLEG): container finished" podID="a00a8a3b-f3da-443c-aaa1-99dc90c2a3d3" containerID="1469a7c2df6a943239c6af72691e1165b4429eea737859aaeaf5dcf763f00a65" exitCode=0
Sep 08 16:59:46.998873 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:46.998469    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b15ca-h4dwk" event={"ID":"a00a8a3b-f3da-443c-aaa1-99dc90c2a3d3","Type":"ContainerDied","Data":"1469a7c2df6a943239c6af72691e1165b4429eea737859aaeaf5dcf763f00a65"}
Sep 08 16:59:46.998873 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:46.998534    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b15ca-h4dwk" event={"ID":"a00a8a3b-f3da-443c-aaa1-99dc90c2a3d3","Type":"ContainerStarted","Data":"8200bfa1017fd07314a5790581bb4819c92b1ce2d1b8d9af7468023e3c2aee6d"}
Sep 08 16:59:48.135389 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:48.135361    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b15ca-h4dwk"
Sep 08 16:59:48.309897 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:48.309782    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00a8a3b-f3da-443c-aaa1-99dc90c2a3d3-maas-api-service-ca\") pod \"a00a8a3b-f3da-443c-aaa1-99dc90c2a3d3\" (UID: \"a00a8a3b-f3da-443c-aaa1-99dc90c2a3d3\") "
Sep 08 16:59:48.310165 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:48.310143    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a00a8a3b-f3da-443c-aaa1-99dc90c2a3d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a00a8a3b-f3da-443c-aaa1-99dc90c2a3d3" (UID: "a00a8a3b-f3da-443c-aaa1-99dc90c2a3d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:59:48.410751 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:48.410712    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00a8a3b-f3da-443c-aaa1-99dc90c2a3d3-maas-api-service-ca\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:59:49.010001 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:49.009966    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b15ca-h4dwk"
Sep 08 16:59:49.010211 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:49.009960    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b15ca-h4dwk" event={"ID":"a00a8a3b-f3da-443c-aaa1-99dc90c2a3d3","Type":"ContainerDied","Data":"8200bfa1017fd07314a5790581bb4819c92b1ce2d1b8d9af7468023e3c2aee6d"}
Sep 08 16:59:49.010211 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:49.010145    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8200bfa1017fd07314a5790581bb4819c92b1ce2d1b8d9af7468023e3c2aee6d"
Sep 08 16:59:56.128385 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:56.128350    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk"]
Sep 08 16:59:56.128940 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:56.128572    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk" podUID="78146cde-eba7-4257-b601-479c516df11c" containerName="payload-processing" containerID="cri-o://a4f4f9bcdae310f9965f9e9632bd8211f3910137e1f4dd41cd4033f462e85e58" gracePeriod=30
Sep 08 16:59:56.135067 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:56.135030    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5"]
Sep 08 16:59:56.135364 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:56.135289    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5" podUID="55b25b67-ec47-49db-8f49-4836d3a35d01" containerName="payload-pre-processing" containerID="cri-o://2cacdfdd576bbfd7fe20d80c331f92a2d3c378a2e269dab429fbf8b9f0e3030b" gracePeriod=30
Sep 08 16:59:56.308941 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:56.308903    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b15ca-h4dwk"]
Sep 08 16:59:56.312326 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:56.312290    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6b15ca-h4dwk"]
Sep 08 16:59:56.349700 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:56.349657    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a00a8a3b-f3da-443c-aaa1-99dc90c2a3d3" path="/var/lib/kubelet/pods/a00a8a3b-f3da-443c-aaa1-99dc90c2a3d3/volumes"
Sep 08 16:59:56.391515 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:56.391451    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk"
Sep 08 16:59:56.423537 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:56.423507    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5"
Sep 08 16:59:56.579852 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:56.579793    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55b25b67-ec47-49db-8f49-4836d3a35d01-plugins-config-volume\") pod \"55b25b67-ec47-49db-8f49-4836d3a35d01\" (UID: \"55b25b67-ec47-49db-8f49-4836d3a35d01\") "
Sep 08 16:59:56.580033 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:56.579932    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxcvd\" (UniqueName: \"kubernetes.io/projected/55b25b67-ec47-49db-8f49-4836d3a35d01-kube-api-access-fxcvd\") pod \"55b25b67-ec47-49db-8f49-4836d3a35d01\" (UID: \"55b25b67-ec47-49db-8f49-4836d3a35d01\") "
Sep 08 16:59:56.580033 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:56.579978    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfwhx\" (UniqueName: \"kubernetes.io/projected/78146cde-eba7-4257-b601-479c516df11c-kube-api-access-lfwhx\") pod \"78146cde-eba7-4257-b601-479c516df11c\" (UID: \"78146cde-eba7-4257-b601-479c516df11c\") "
Sep 08 16:59:56.580033 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:56.579999    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78146cde-eba7-4257-b601-479c516df11c-plugins-config-volume\") pod \"78146cde-eba7-4257-b601-479c516df11c\" (UID: \"78146cde-eba7-4257-b601-479c516df11c\") "
Sep 08 16:59:56.580274 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:56.580250    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55b25b67-ec47-49db-8f49-4836d3a35d01-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "55b25b67-ec47-49db-8f49-4836d3a35d01" (UID: "55b25b67-ec47-49db-8f49-4836d3a35d01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:59:56.580449 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:56.580424    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78146cde-eba7-4257-b601-479c516df11c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "78146cde-eba7-4257-b601-479c516df11c" (UID: "78146cde-eba7-4257-b601-479c516df11c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:59:56.582313 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:56.582285    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b25b67-ec47-49db-8f49-4836d3a35d01-kube-api-access-fxcvd" (OuterVolumeSpecName: "kube-api-access-fxcvd") pod "55b25b67-ec47-49db-8f49-4836d3a35d01" (UID: "55b25b67-ec47-49db-8f49-4836d3a35d01"). InnerVolumeSpecName "kube-api-access-fxcvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:59:56.582395 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:56.582296    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78146cde-eba7-4257-b601-479c516df11c-kube-api-access-lfwhx" (OuterVolumeSpecName: "kube-api-access-lfwhx") pod "78146cde-eba7-4257-b601-479c516df11c" (UID: "78146cde-eba7-4257-b601-479c516df11c"). InnerVolumeSpecName "kube-api-access-lfwhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:59:56.680804 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:56.680759    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxcvd\" (UniqueName: \"kubernetes.io/projected/55b25b67-ec47-49db-8f49-4836d3a35d01-kube-api-access-fxcvd\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:59:56.680804 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:56.680798    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfwhx\" (UniqueName: \"kubernetes.io/projected/78146cde-eba7-4257-b601-479c516df11c-kube-api-access-lfwhx\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:59:56.680804 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:56.680807    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78146cde-eba7-4257-b601-479c516df11c-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:59:56.680804 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:56.680815    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55b25b67-ec47-49db-8f49-4836d3a35d01-plugins-config-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 16:59:57.053196 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:57.053103    2557 generic.go:356] "Generic (PLEG): container finished" podID="78146cde-eba7-4257-b601-479c516df11c" containerID="a4f4f9bcdae310f9965f9e9632bd8211f3910137e1f4dd41cd4033f462e85e58" exitCode=0
Sep 08 16:59:57.053196 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:57.053154    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk"
Sep 08 16:59:57.053431 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:57.053198    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk" event={"ID":"78146cde-eba7-4257-b601-479c516df11c","Type":"ContainerDied","Data":"a4f4f9bcdae310f9965f9e9632bd8211f3910137e1f4dd41cd4033f462e85e58"}
Sep 08 16:59:57.053431 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:57.053249    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk" event={"ID":"78146cde-eba7-4257-b601-479c516df11c","Type":"ContainerDied","Data":"d0ed89c7dd9a1a71ad562a1aafc3ff57f746670f74ab1a9fd02426a352fc11dc"}
Sep 08 16:59:57.053431 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:57.053271    2557 scope.go:117] "RemoveContainer" containerID="a4f4f9bcdae310f9965f9e9632bd8211f3910137e1f4dd41cd4033f462e85e58"
Sep 08 16:59:57.054883 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:57.054853    2557 generic.go:356] "Generic (PLEG): container finished" podID="55b25b67-ec47-49db-8f49-4836d3a35d01" containerID="2cacdfdd576bbfd7fe20d80c331f92a2d3c378a2e269dab429fbf8b9f0e3030b" exitCode=0
Sep 08 16:59:57.055016 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:57.054905    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5"
Sep 08 16:59:57.055016 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:57.054941    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5" event={"ID":"55b25b67-ec47-49db-8f49-4836d3a35d01","Type":"ContainerDied","Data":"2cacdfdd576bbfd7fe20d80c331f92a2d3c378a2e269dab429fbf8b9f0e3030b"}
Sep 08 16:59:57.055016 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:57.054984    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5" event={"ID":"55b25b67-ec47-49db-8f49-4836d3a35d01","Type":"ContainerDied","Data":"c3b567c8eb0be064ca31b467d714e98d244ae4954789231bd81c40394d51ad56"}
Sep 08 16:59:57.064041 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:57.064014    2557 scope.go:117] "RemoveContainer" containerID="a4f4f9bcdae310f9965f9e9632bd8211f3910137e1f4dd41cd4033f462e85e58"
Sep 08 16:59:57.064426 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:59:57.064403    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4f4f9bcdae310f9965f9e9632bd8211f3910137e1f4dd41cd4033f462e85e58\": container with ID starting with a4f4f9bcdae310f9965f9e9632bd8211f3910137e1f4dd41cd4033f462e85e58 not found: ID does not exist" containerID="a4f4f9bcdae310f9965f9e9632bd8211f3910137e1f4dd41cd4033f462e85e58"
Sep 08 16:59:57.064549 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:57.064432    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4f4f9bcdae310f9965f9e9632bd8211f3910137e1f4dd41cd4033f462e85e58"} err="failed to get container status \"a4f4f9bcdae310f9965f9e9632bd8211f3910137e1f4dd41cd4033f462e85e58\": rpc error: code = NotFound desc = could not find container \"a4f4f9bcdae310f9965f9e9632bd8211f3910137e1f4dd41cd4033f462e85e58\": container with ID starting with a4f4f9bcdae310f9965f9e9632bd8211f3910137e1f4dd41cd4033f462e85e58 not found: ID does not exist"
Sep 08 16:59:57.064549 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:57.064455    2557 scope.go:117] "RemoveContainer" containerID="2cacdfdd576bbfd7fe20d80c331f92a2d3c378a2e269dab429fbf8b9f0e3030b"
Sep 08 16:59:57.075587 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:57.075560    2557 scope.go:117] "RemoveContainer" containerID="2cacdfdd576bbfd7fe20d80c331f92a2d3c378a2e269dab429fbf8b9f0e3030b"
Sep 08 16:59:57.075974 ip-10-0-139-51 kubenswrapper[2557]: E0908 16:59:57.075954    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cacdfdd576bbfd7fe20d80c331f92a2d3c378a2e269dab429fbf8b9f0e3030b\": container with ID starting with 2cacdfdd576bbfd7fe20d80c331f92a2d3c378a2e269dab429fbf8b9f0e3030b not found: ID does not exist" containerID="2cacdfdd576bbfd7fe20d80c331f92a2d3c378a2e269dab429fbf8b9f0e3030b"
Sep 08 16:59:57.076035 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:57.075985    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cacdfdd576bbfd7fe20d80c331f92a2d3c378a2e269dab429fbf8b9f0e3030b"} err="failed to get container status \"2cacdfdd576bbfd7fe20d80c331f92a2d3c378a2e269dab429fbf8b9f0e3030b\": rpc error: code = NotFound desc = could not find container \"2cacdfdd576bbfd7fe20d80c331f92a2d3c378a2e269dab429fbf8b9f0e3030b\": container with ID starting with 2cacdfdd576bbfd7fe20d80c331f92a2d3c378a2e269dab429fbf8b9f0e3030b not found: ID does not exist"
Sep 08 16:59:57.090014 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:57.089973    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5"]
Sep 08 16:59:57.097731 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:57.097697    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b15ca-75448c6754-b8qs5"]
Sep 08 16:59:57.124664 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:57.124625    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk"]
Sep 08 16:59:57.135161 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:57.135122    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6b15ca-696459f597-sjpnk"]
Sep 08 16:59:58.345182 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:58.345150    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55b25b67-ec47-49db-8f49-4836d3a35d01" path="/var/lib/kubelet/pods/55b25b67-ec47-49db-8f49-4836d3a35d01/volumes"
Sep 08 16:59:58.345581 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:58.345492    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78146cde-eba7-4257-b601-479c516df11c" path="/var/lib/kubelet/pods/78146cde-eba7-4257-b601-479c516df11c/volumes"
Sep 08 16:59:59.039388 ip-10-0-139-51 kubenswrapper[2557]: I0908 16:59:59.039352    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 17:00:00.139577 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:00.139542    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814780-qh5ft"]
Sep 08 17:00:00.139996 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:00.139883    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55b25b67-ec47-49db-8f49-4836d3a35d01" containerName="payload-pre-processing"
Sep 08 17:00:00.139996 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:00.139897    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="55b25b67-ec47-49db-8f49-4836d3a35d01" containerName="payload-pre-processing"
Sep 08 17:00:00.139996 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:00.139917    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78146cde-eba7-4257-b601-479c516df11c" containerName="payload-processing"
Sep 08 17:00:00.139996 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:00.139923    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="78146cde-eba7-4257-b601-479c516df11c" containerName="payload-processing"
Sep 08 17:00:00.139996 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:00.139938    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a00a8a3b-f3da-443c-aaa1-99dc90c2a3d3" containerName="revoke-keys"
Sep 08 17:00:00.139996 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:00.139947    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a00a8a3b-f3da-443c-aaa1-99dc90c2a3d3" containerName="revoke-keys"
Sep 08 17:00:00.140186 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:00.140004    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="55b25b67-ec47-49db-8f49-4836d3a35d01" containerName="payload-pre-processing"
Sep 08 17:00:00.140186 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:00.140015    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="78146cde-eba7-4257-b601-479c516df11c" containerName="payload-processing"
Sep 08 17:00:00.140186 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:00.140022    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a00a8a3b-f3da-443c-aaa1-99dc90c2a3d3" containerName="revoke-keys"
Sep 08 17:00:00.144198 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:00.144175    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814780-qh5ft"
Sep 08 17:00:00.147395 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:00.147373    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-886jf\""
Sep 08 17:00:00.151241 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:00.151210    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814780-qh5ft"]
Sep 08 17:00:00.155630 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:00.155601    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814780-qh5ft"
Sep 08 17:00:00.437916 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:00.437879    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814780-qh5ft"]
Sep 08 17:00:00.441915 ip-10-0-139-51 kubenswrapper[2557]: W0908 17:00:00.441884    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0498b1a9_1215_4adf_900d_81480986051f.slice/crio-89425303c1481244e31b6f1fde87f71521b11591568c149d9630236082f87861 WatchSource:0}: Error finding container 89425303c1481244e31b6f1fde87f71521b11591568c149d9630236082f87861: Status 404 returned error can't find the container with id 89425303c1481244e31b6f1fde87f71521b11591568c149d9630236082f87861
Sep 08 17:00:01.080963 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:01.080871    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814780-qh5ft" event={"ID":"0498b1a9-1215-4adf-900d-81480986051f","Type":"ContainerStarted","Data":"473dc97233a460b2a748950b7760ccef6bcece295870562f217c20493974bb73"}
Sep 08 17:00:01.080963 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:01.080920    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814780-qh5ft" event={"ID":"0498b1a9-1215-4adf-900d-81480986051f","Type":"ContainerStarted","Data":"89425303c1481244e31b6f1fde87f71521b11591568c149d9630236082f87861"}
Sep 08 17:00:01.097878 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:01.097794    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814780-qh5ft" podStartSLOduration=1.097773916 podStartE2EDuration="1.097773916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:00:01.096316483 +0000 UTC m=+1785.259287944" watchObservedRunningTime="2026-09-08 17:00:01.097773916 +0000 UTC m=+1785.260745364"
Sep 08 17:00:02.089106 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:02.089062    2557 generic.go:356] "Generic (PLEG): container finished" podID="0498b1a9-1215-4adf-900d-81480986051f" containerID="473dc97233a460b2a748950b7760ccef6bcece295870562f217c20493974bb73" exitCode=0
Sep 08 17:00:02.089536 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:02.089123    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814780-qh5ft" event={"ID":"0498b1a9-1215-4adf-900d-81480986051f","Type":"ContainerDied","Data":"473dc97233a460b2a748950b7760ccef6bcece295870562f217c20493974bb73"}
Sep 08 17:00:03.235721 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:03.235694    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814780-qh5ft"
Sep 08 17:00:04.102496 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:04.102465    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814780-qh5ft"
Sep 08 17:00:04.102655 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:04.102494    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814780-qh5ft" event={"ID":"0498b1a9-1215-4adf-900d-81480986051f","Type":"ContainerDied","Data":"89425303c1481244e31b6f1fde87f71521b11591568c149d9630236082f87861"}
Sep 08 17:00:04.102655 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:04.102541    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89425303c1481244e31b6f1fde87f71521b11591568c149d9630236082f87861"
Sep 08 17:00:08.040542 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:08.040456    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 17:00:16.764488 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:00:16.764454    2557 scope.go:117] "RemoveContainer" containerID="b14f5344d69e17cae6a52058e85fe477a736fd01f76baaab754514549ac45fe0"
Sep 08 17:01:16.808570 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:01:16.808535    2557 scope.go:117] "RemoveContainer" containerID="84478ccb29a9920f310ba2cdf1d5413b71f7c6260df2df2f3766540e748d1091"
Sep 08 17:01:16.816998 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:01:16.816977    2557 scope.go:117] "RemoveContainer" containerID="4babb6ab70cf3ae0b53abafe25fb197720c9c47257a34f57085201d73c352470"
Sep 08 17:01:28.742191 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:01:28.742146    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 17:01:52.949537 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:01:52.949454    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 17:02:00.744978 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:00.744936    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 17:02:09.873298 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:09.873257    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 17:02:16.839080 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:16.839040    2557 scope.go:117] "RemoveContainer" containerID="87a2342345641e3531fc573b39cf965c4ee77fe8e590c566f33cc3cf2cc25316"
Sep 08 17:02:16.847738 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:16.847715    2557 scope.go:117] "RemoveContainer" containerID="beef2f6802fa11ac3ba11e406b919a3a5d33bc833080cb35dfdbdf89301d0b50"
Sep 08 17:02:28.949038 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:28.949003    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 17:02:35.545052 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:35.545012    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 17:02:38.631239 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:38.631204    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ff8f8"]
Sep 08 17:02:38.631674 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:38.631409    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ff8f8" podUID="50c14b2c-7734-4370-bae2-c0ebbccd57f0" containerName="manager" containerID="cri-o://33736d982c82f0360299ced71998f81eb18b440de5829ac946d35fe8a988227e" gracePeriod=10
Sep 08 17:02:39.591019 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:39.590988    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ff8f8"
Sep 08 17:02:39.755879 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:39.755737    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50c14b2c-7734-4370-bae2-c0ebbccd57f0-extensions-socket-volume\") pod \"50c14b2c-7734-4370-bae2-c0ebbccd57f0\" (UID: \"50c14b2c-7734-4370-bae2-c0ebbccd57f0\") "
Sep 08 17:02:39.755879 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:39.755803    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ggfb\" (UniqueName: \"kubernetes.io/projected/50c14b2c-7734-4370-bae2-c0ebbccd57f0-kube-api-access-7ggfb\") pod \"50c14b2c-7734-4370-bae2-c0ebbccd57f0\" (UID: \"50c14b2c-7734-4370-bae2-c0ebbccd57f0\") "
Sep 08 17:02:39.756281 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:39.756148    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c14b2c-7734-4370-bae2-c0ebbccd57f0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "50c14b2c-7734-4370-bae2-c0ebbccd57f0" (UID: "50c14b2c-7734-4370-bae2-c0ebbccd57f0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:02:39.758009 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:39.757976    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c14b2c-7734-4370-bae2-c0ebbccd57f0-kube-api-access-7ggfb" (OuterVolumeSpecName: "kube-api-access-7ggfb") pod "50c14b2c-7734-4370-bae2-c0ebbccd57f0" (UID: "50c14b2c-7734-4370-bae2-c0ebbccd57f0"). InnerVolumeSpecName "kube-api-access-7ggfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:02:39.857206 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:39.857149    2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50c14b2c-7734-4370-bae2-c0ebbccd57f0-extensions-socket-volume\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 17:02:39.857206 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:39.857199    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ggfb\" (UniqueName: \"kubernetes.io/projected/50c14b2c-7734-4370-bae2-c0ebbccd57f0-kube-api-access-7ggfb\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 17:02:39.887915 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:39.887880    2557 generic.go:356] "Generic (PLEG): container finished" podID="50c14b2c-7734-4370-bae2-c0ebbccd57f0" containerID="33736d982c82f0360299ced71998f81eb18b440de5829ac946d35fe8a988227e" exitCode=0
Sep 08 17:02:39.888101 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:39.887924    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ff8f8" event={"ID":"50c14b2c-7734-4370-bae2-c0ebbccd57f0","Type":"ContainerDied","Data":"33736d982c82f0360299ced71998f81eb18b440de5829ac946d35fe8a988227e"}
Sep 08 17:02:39.888101 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:39.887929    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ff8f8"
Sep 08 17:02:39.888101 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:39.887969    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ff8f8" event={"ID":"50c14b2c-7734-4370-bae2-c0ebbccd57f0","Type":"ContainerDied","Data":"61c4e4bb2c37c6b017ce4079f10cd532d6d5bf55c9796dba5cf1510ee8bed10c"}
Sep 08 17:02:39.888101 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:39.887993    2557 scope.go:117] "RemoveContainer" containerID="33736d982c82f0360299ced71998f81eb18b440de5829ac946d35fe8a988227e"
Sep 08 17:02:39.897631 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:39.897610    2557 scope.go:117] "RemoveContainer" containerID="33736d982c82f0360299ced71998f81eb18b440de5829ac946d35fe8a988227e"
Sep 08 17:02:39.898088 ip-10-0-139-51 kubenswrapper[2557]: E0908 17:02:39.898068    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33736d982c82f0360299ced71998f81eb18b440de5829ac946d35fe8a988227e\": container with ID starting with 33736d982c82f0360299ced71998f81eb18b440de5829ac946d35fe8a988227e not found: ID does not exist" containerID="33736d982c82f0360299ced71998f81eb18b440de5829ac946d35fe8a988227e"
Sep 08 17:02:39.898172 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:39.898105    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33736d982c82f0360299ced71998f81eb18b440de5829ac946d35fe8a988227e"} err="failed to get container status \"33736d982c82f0360299ced71998f81eb18b440de5829ac946d35fe8a988227e\": rpc error: code = NotFound desc = could not find container \"33736d982c82f0360299ced71998f81eb18b440de5829ac946d35fe8a988227e\": container with ID starting with 33736d982c82f0360299ced71998f81eb18b440de5829ac946d35fe8a988227e not found: ID does not exist"
Sep 08 17:02:39.912066 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:39.912022    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ff8f8"]
Sep 08 17:02:39.917445 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:39.917407    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ff8f8"]
Sep 08 17:02:40.344626 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:02:40.344590    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50c14b2c-7734-4370-bae2-c0ebbccd57f0" path="/var/lib/kubelet/pods/50c14b2c-7734-4370-bae2-c0ebbccd57f0/volumes"
Sep 08 17:03:39.352461 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:03:39.352427    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gfgpt"]
Sep 08 17:03:39.352981 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:03:39.352734    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50c14b2c-7734-4370-bae2-c0ebbccd57f0" containerName="manager"
Sep 08 17:03:39.352981 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:03:39.352745    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="50c14b2c-7734-4370-bae2-c0ebbccd57f0" containerName="manager"
Sep 08 17:03:39.352981 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:03:39.352764    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0498b1a9-1215-4adf-900d-81480986051f" containerName="cleanup"
Sep 08 17:03:39.352981 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:03:39.352771    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0498b1a9-1215-4adf-900d-81480986051f" containerName="cleanup"
Sep 08 17:03:39.352981 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:03:39.352823    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0498b1a9-1215-4adf-900d-81480986051f" containerName="cleanup"
Sep 08 17:03:39.352981 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:03:39.352853    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="50c14b2c-7734-4370-bae2-c0ebbccd57f0" containerName="manager"
Sep 08 17:03:39.355745 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:03:39.355726    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gfgpt"
Sep 08 17:03:39.358998 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:03:39.358970    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ts7rd\""
Sep 08 17:03:39.377546 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:03:39.377515    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gfgpt"]
Sep 08 17:03:39.418753 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:03:39.418703    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq5sx\" (UniqueName: \"kubernetes.io/projected/db6bc462-753b-4f07-8bc8-95cdadb4a611-kube-api-access-sq5sx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gfgpt\" (UID: \"db6bc462-753b-4f07-8bc8-95cdadb4a611\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gfgpt"
Sep 08 17:03:39.418970 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:03:39.418869    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db6bc462-753b-4f07-8bc8-95cdadb4a611-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gfgpt\" (UID: \"db6bc462-753b-4f07-8bc8-95cdadb4a611\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gfgpt"
Sep 08 17:03:39.519910 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:03:39.519866    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db6bc462-753b-4f07-8bc8-95cdadb4a611-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gfgpt\" (UID: \"db6bc462-753b-4f07-8bc8-95cdadb4a611\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gfgpt"
Sep 08 17:03:39.520101 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:03:39.519928    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq5sx\" (UniqueName: \"kubernetes.io/projected/db6bc462-753b-4f07-8bc8-95cdadb4a611-kube-api-access-sq5sx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gfgpt\" (UID: \"db6bc462-753b-4f07-8bc8-95cdadb4a611\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gfgpt"
Sep 08 17:03:39.520285 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:03:39.520263    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db6bc462-753b-4f07-8bc8-95cdadb4a611-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gfgpt\" (UID: \"db6bc462-753b-4f07-8bc8-95cdadb4a611\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gfgpt"
Sep 08 17:03:39.531734 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:03:39.531696    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq5sx\" (UniqueName: \"kubernetes.io/projected/db6bc462-753b-4f07-8bc8-95cdadb4a611-kube-api-access-sq5sx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gfgpt\" (UID: \"db6bc462-753b-4f07-8bc8-95cdadb4a611\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gfgpt"
Sep 08 17:03:39.665276 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:03:39.665231    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gfgpt"
Sep 08 17:03:39.990210 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:03:39.990179    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gfgpt"]
Sep 08 17:03:39.990616 ip-10-0-139-51 kubenswrapper[2557]: W0908 17:03:39.990592    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb6bc462_753b_4f07_8bc8_95cdadb4a611.slice/crio-897fe6bcc8756861f3cb27536fd757be2933657d8683f433d6dd61065525ca4f WatchSource:0}: Error finding container 897fe6bcc8756861f3cb27536fd757be2933657d8683f433d6dd61065525ca4f: Status 404 returned error can't find the container with id 897fe6bcc8756861f3cb27536fd757be2933657d8683f433d6dd61065525ca4f
Sep 08 17:03:40.184074 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:03:40.184038    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gfgpt" event={"ID":"db6bc462-753b-4f07-8bc8-95cdadb4a611","Type":"ContainerStarted","Data":"d8bcef771e34c7ce34dda9d5599b5acd7e2da32ca05152d85a369c263e1620bc"}
Sep 08 17:03:40.184261 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:03:40.184101    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gfgpt" event={"ID":"db6bc462-753b-4f07-8bc8-95cdadb4a611","Type":"ContainerStarted","Data":"897fe6bcc8756861f3cb27536fd757be2933657d8683f433d6dd61065525ca4f"}
Sep 08 17:03:40.184261 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:03:40.184217    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gfgpt"
Sep 08 17:03:40.235353 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:03:40.235294    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gfgpt" podStartSLOduration=1.235277283 podStartE2EDuration="1.235277283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:03:40.232926684 +0000 UTC m=+2004.395898132" watchObservedRunningTime="2026-09-08 17:03:40.235277283 +0000 UTC m=+2004.398248731"
Sep 08 17:03:51.191213 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:03:51.191171    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gfgpt"
Sep 08 17:04:04.874052 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:04.874014    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 17:04:14.038159 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:14.038116    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 17:04:16.884189 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:16.884153    2557 scope.go:117] "RemoveContainer" containerID="39cd260d4fc87f0348746424a03a1e4a575fe4ebbd1c28716189b77c4eb3dc2b"
Sep 08 17:04:16.892718 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:16.892696    2557 scope.go:117] "RemoveContainer" containerID="0b92a7efa0c5664768e606e1083b94ea2c46f6afd2ed3de9ca4fa5fc2966a277"
Sep 08 17:04:18.560890 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:18.560851    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"]
Sep 08 17:04:18.565552 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:18.565525    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:04:18.568587 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:18.568558    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-15d29298-kserve-self-signed-certs\""
Sep 08 17:04:18.578081 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:18.578048    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"]
Sep 08 17:04:18.658618 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:18.658582    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-dshm\") pod \"e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m\" (UID: \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\") " pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:04:18.658811 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:18.658634    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-home\") pod \"e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m\" (UID: \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\") " pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:04:18.658811 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:18.658675    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-tls-certs\") pod \"e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m\" (UID: \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\") " pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:04:18.658811 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:18.658703    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-tmp-dir\") pod \"e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m\" (UID: \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\") " pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:04:18.658811 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:18.658795    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfdpg\" (UniqueName: \"kubernetes.io/projected/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-kube-api-access-nfdpg\") pod \"e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m\" (UID: \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\") " pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:04:18.658992 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:18.658851    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-model-cache\") pod \"e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m\" (UID: \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\") " pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:04:18.759513 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:18.759472    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-dshm\") pod \"e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m\" (UID: \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\") " pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:04:18.759513 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:18.759523    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-home\") pod \"e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m\" (UID: \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\") " pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:04:18.759753 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:18.759553    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-tls-certs\") pod \"e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m\" (UID: \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\") " pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:04:18.759753 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:18.759581    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-tmp-dir\") pod \"e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m\" (UID: \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\") " pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:04:18.759753 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:18.759606    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfdpg\" (UniqueName: \"kubernetes.io/projected/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-kube-api-access-nfdpg\") pod \"e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m\" (UID: \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\") " pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:04:18.759753 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:18.759628    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-model-cache\") pod \"e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m\" (UID: \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\") " pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:04:18.760065 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:18.760036    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-home\") pod \"e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m\" (UID: \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\") " pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:04:18.760065 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:18.760055    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-model-cache\") pod \"e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m\" (UID: \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\") " pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:04:18.760190 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:18.760093    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-tmp-dir\") pod \"e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m\" (UID: \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\") " pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:04:18.761946 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:18.761923    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-dshm\") pod \"e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m\" (UID: \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\") " pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:04:18.762226 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:18.762206    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-tls-certs\") pod \"e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m\" (UID: \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\") " pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:04:18.770141 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:18.770112    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfdpg\" (UniqueName: \"kubernetes.io/projected/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-kube-api-access-nfdpg\") pod \"e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m\" (UID: \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\") " pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:04:18.875289 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:18.875187    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:04:19.157364 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:19.157334    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"]
Sep 08 17:04:19.158862 ip-10-0-139-51 kubenswrapper[2557]: W0908 17:04:19.158814    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff2aca5b_210e_4a0e_af0d_17a402ba4f73.slice/crio-6cc79c566cd5a43a57ad291104addf5cf5f809363645a4baca132c7c2599075b WatchSource:0}: Error finding container 6cc79c566cd5a43a57ad291104addf5cf5f809363645a4baca132c7c2599075b: Status 404 returned error can't find the container with id 6cc79c566cd5a43a57ad291104addf5cf5f809363645a4baca132c7c2599075b
Sep 08 17:04:19.379220 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:19.379180    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m" event={"ID":"ff2aca5b-210e-4a0e-af0d-17a402ba4f73","Type":"ContainerStarted","Data":"493588125a60cb4dc9e520e7307460246161a241cb5fef68500d0a1ccff75a1e"}
Sep 08 17:04:19.379450 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:19.379241    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m" event={"ID":"ff2aca5b-210e-4a0e-af0d-17a402ba4f73","Type":"ContainerStarted","Data":"6cc79c566cd5a43a57ad291104addf5cf5f809363645a4baca132c7c2599075b"}
Sep 08 17:04:19.403485 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:19.403424    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m" podStartSLOduration=1.403403316 podStartE2EDuration="1.403403316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:04:19.402694648 +0000 UTC m=+2043.565666097" watchObservedRunningTime="2026-09-08 17:04:19.403403316 +0000 UTC m=+2043.566375147"
Sep 08 17:04:21.374258 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:21.374219    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 17:04:28.876240 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:28.876197    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:04:28.876647 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:28.876252    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:04:28.889298 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:28.889262    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:04:29.438364 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:29.438328    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:04:50.658157 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:50.658074    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"]
Sep 08 17:04:50.661388 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:50.661363    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:04:50.664961 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:50.664936    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-15d29298-kserve-self-signed-certs\""
Sep 08 17:04:50.677369 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:50.677307    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"]
Sep 08 17:04:50.730237 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:50.730191    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7d02cc4-93c2-45c3-a357-a8e0ab494395-dshm\") pod \"e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t\" (UID: \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\") " pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:04:50.730419 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:50.730245    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpqld\" (UniqueName: \"kubernetes.io/projected/d7d02cc4-93c2-45c3-a357-a8e0ab494395-kube-api-access-dpqld\") pod \"e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t\" (UID: \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\") " pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:04:50.730419 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:50.730280    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d02cc4-93c2-45c3-a357-a8e0ab494395-tls-certs\") pod \"e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t\" (UID: \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\") " pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:04:50.730419 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:50.730385    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7d02cc4-93c2-45c3-a357-a8e0ab494395-model-cache\") pod \"e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t\" (UID: \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\") " pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:04:50.730419 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:50.730412    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7d02cc4-93c2-45c3-a357-a8e0ab494395-tmp-dir\") pod \"e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t\" (UID: \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\") " pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:04:50.730625 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:50.730441    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7d02cc4-93c2-45c3-a357-a8e0ab494395-home\") pod \"e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t\" (UID: \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\") " pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:04:50.831334 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:50.831288    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7d02cc4-93c2-45c3-a357-a8e0ab494395-model-cache\") pod \"e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t\" (UID: \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\") " pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:04:50.831334 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:50.831331    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7d02cc4-93c2-45c3-a357-a8e0ab494395-tmp-dir\") pod \"e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t\" (UID: \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\") " pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:04:50.831600 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:50.831350    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7d02cc4-93c2-45c3-a357-a8e0ab494395-home\") pod \"e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t\" (UID: \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\") " pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:04:50.831600 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:50.831372    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7d02cc4-93c2-45c3-a357-a8e0ab494395-dshm\") pod \"e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t\" (UID: \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\") " pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:04:50.831600 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:50.831408    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpqld\" (UniqueName: \"kubernetes.io/projected/d7d02cc4-93c2-45c3-a357-a8e0ab494395-kube-api-access-dpqld\") pod \"e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t\" (UID: \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\") " pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:04:50.831600 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:50.831442    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d02cc4-93c2-45c3-a357-a8e0ab494395-tls-certs\") pod \"e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t\" (UID: \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\") " pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:04:50.831812 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:50.831757    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7d02cc4-93c2-45c3-a357-a8e0ab494395-model-cache\") pod \"e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t\" (UID: \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\") " pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:04:50.831915 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:50.831817    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7d02cc4-93c2-45c3-a357-a8e0ab494395-tmp-dir\") pod \"e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t\" (UID: \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\") " pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:04:50.831915 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:50.831889    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7d02cc4-93c2-45c3-a357-a8e0ab494395-home\") pod \"e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t\" (UID: \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\") " pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:04:50.833971 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:50.833951    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7d02cc4-93c2-45c3-a357-a8e0ab494395-dshm\") pod \"e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t\" (UID: \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\") " pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:04:50.834266 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:50.834245    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d02cc4-93c2-45c3-a357-a8e0ab494395-tls-certs\") pod \"e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t\" (UID: \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\") " pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:04:50.841281 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:50.841255    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpqld\" (UniqueName: \"kubernetes.io/projected/d7d02cc4-93c2-45c3-a357-a8e0ab494395-kube-api-access-dpqld\") pod \"e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t\" (UID: \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\") " pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:04:50.971805 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:50.971758    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:04:51.247938 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:51.247911    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"]
Sep 08 17:04:51.250203 ip-10-0-139-51 kubenswrapper[2557]: W0908 17:04:51.250173    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7d02cc4_93c2_45c3_a357_a8e0ab494395.slice/crio-367b7fc7e3ecc8e56ed8627313439409da3f101ad7fd36dbd71d498cde1e6fea WatchSource:0}: Error finding container 367b7fc7e3ecc8e56ed8627313439409da3f101ad7fd36dbd71d498cde1e6fea: Status 404 returned error can't find the container with id 367b7fc7e3ecc8e56ed8627313439409da3f101ad7fd36dbd71d498cde1e6fea
Sep 08 17:04:51.532066 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:51.531965    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t" event={"ID":"d7d02cc4-93c2-45c3-a357-a8e0ab494395","Type":"ContainerStarted","Data":"7e5535a8c0b13c6ebce8bda842c507e7e6f02e7e2e93e2f953b0c38e7e7e8644"}
Sep 08 17:04:51.532066 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:51.532013    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t" event={"ID":"d7d02cc4-93c2-45c3-a357-a8e0ab494395","Type":"ContainerStarted","Data":"367b7fc7e3ecc8e56ed8627313439409da3f101ad7fd36dbd71d498cde1e6fea"}
Sep 08 17:04:51.552025 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:51.551966    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t" podStartSLOduration=1.551928577 podStartE2EDuration="1.551928577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:04:51.550768727 +0000 UTC m=+2075.713740185" watchObservedRunningTime="2026-09-08 17:04:51.551928577 +0000 UTC m=+2075.714900024"
Sep 08 17:04:53.495499 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:04:53.495458    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 17:05:00.972641 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:00.972595    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:05:00.973068 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:00.972661    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:05:00.985913 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:00.985881    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:05:01.591946 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:01.591911    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:05:14.527911 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:14.527872    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"]
Sep 08 17:05:16.634086 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:16.634049    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 17:05:16.858130 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:16.858097    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"]
Sep 08 17:05:16.917011 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:16.916984    2557 scope.go:117] "RemoveContainer" containerID="a6698c86034fd643c87f7811a26f90fd87193524e990e948e943237a00a8b035"
Sep 08 17:05:24.736151 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:24.736114    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 17:05:29.548402 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:29.548362    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m" podUID="ff2aca5b-210e-4a0e-af0d-17a402ba4f73" containerName="main" containerID="cri-o://493588125a60cb4dc9e520e7307460246161a241cb5fef68500d0a1ccff75a1e" gracePeriod=45
Sep 08 17:05:29.715340 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:29.715304    2557 generic.go:356] "Generic (PLEG): container finished" podID="ff2aca5b-210e-4a0e-af0d-17a402ba4f73" containerID="493588125a60cb4dc9e520e7307460246161a241cb5fef68500d0a1ccff75a1e" exitCode=0
Sep 08 17:05:29.715508 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:29.715370    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m" event={"ID":"ff2aca5b-210e-4a0e-af0d-17a402ba4f73","Type":"ContainerDied","Data":"493588125a60cb4dc9e520e7307460246161a241cb5fef68500d0a1ccff75a1e"}
Sep 08 17:05:29.807322 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:29.807257    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:05:29.866123 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:29.866090    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-tmp-dir\") pod \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\" (UID: \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\") "
Sep 08 17:05:29.866297 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:29.866133    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-tls-certs\") pod \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\" (UID: \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\") "
Sep 08 17:05:29.866297 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:29.866155    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfdpg\" (UniqueName: \"kubernetes.io/projected/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-kube-api-access-nfdpg\") pod \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\" (UID: \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\") "
Sep 08 17:05:29.866297 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:29.866181    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-dshm\") pod \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\" (UID: \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\") "
Sep 08 17:05:29.866297 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:29.866224    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-home\") pod \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\" (UID: \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\") "
Sep 08 17:05:29.866297 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:29.866286    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-model-cache\") pod \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\" (UID: \"ff2aca5b-210e-4a0e-af0d-17a402ba4f73\") "
Sep 08 17:05:29.866577 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:29.866449    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ff2aca5b-210e-4a0e-af0d-17a402ba4f73" (UID: "ff2aca5b-210e-4a0e-af0d-17a402ba4f73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:05:29.866636 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:29.866583    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 17:05:29.866636 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:29.866579    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-home" (OuterVolumeSpecName: "home") pod "ff2aca5b-210e-4a0e-af0d-17a402ba4f73" (UID: "ff2aca5b-210e-4a0e-af0d-17a402ba4f73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:05:29.866712 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:29.866615    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-model-cache" (OuterVolumeSpecName: "model-cache") pod "ff2aca5b-210e-4a0e-af0d-17a402ba4f73" (UID: "ff2aca5b-210e-4a0e-af0d-17a402ba4f73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:05:29.868503 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:29.868476    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-dshm" (OuterVolumeSpecName: "dshm") pod "ff2aca5b-210e-4a0e-af0d-17a402ba4f73" (UID: "ff2aca5b-210e-4a0e-af0d-17a402ba4f73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:05:29.868607 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:29.868542    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-kube-api-access-nfdpg" (OuterVolumeSpecName: "kube-api-access-nfdpg") pod "ff2aca5b-210e-4a0e-af0d-17a402ba4f73" (UID: "ff2aca5b-210e-4a0e-af0d-17a402ba4f73"). InnerVolumeSpecName "kube-api-access-nfdpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:05:29.868607 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:29.868553    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff2aca5b-210e-4a0e-af0d-17a402ba4f73" (UID: "ff2aca5b-210e-4a0e-af0d-17a402ba4f73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 17:05:29.967033 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:29.966990    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 17:05:29.967033 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:29.967024    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 17:05:29.967033 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:29.967036    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 17:05:29.967284 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:29.967048    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 17:05:29.967284 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:29.967059    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfdpg\" (UniqueName: \"kubernetes.io/projected/ff2aca5b-210e-4a0e-af0d-17a402ba4f73-kube-api-access-nfdpg\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 17:05:30.554108 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:30.554073    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 17:05:30.722574 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:30.722542    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"
Sep 08 17:05:30.722574 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:30.722565    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m" event={"ID":"ff2aca5b-210e-4a0e-af0d-17a402ba4f73","Type":"ContainerDied","Data":"6cc79c566cd5a43a57ad291104addf5cf5f809363645a4baca132c7c2599075b"}
Sep 08 17:05:30.722811 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:30.722620    2557 scope.go:117] "RemoveContainer" containerID="493588125a60cb4dc9e520e7307460246161a241cb5fef68500d0a1ccff75a1e"
Sep 08 17:05:30.743912 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:30.743876    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"]
Sep 08 17:05:30.746029 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:30.746001    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-15d29298-kserve-6f8fd8895-sbg2m"]
Sep 08 17:05:31.879368 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:31.879294    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t" podUID="d7d02cc4-93c2-45c3-a357-a8e0ab494395" containerName="main" containerID="cri-o://7e5535a8c0b13c6ebce8bda842c507e7e6f02e7e2e93e2f953b0c38e7e7e8644" gracePeriod=45
Sep 08 17:05:32.140157 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.140086    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:05:32.186263 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.186221    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7d02cc4-93c2-45c3-a357-a8e0ab494395-home\") pod \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\" (UID: \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\") "
Sep 08 17:05:32.186463 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.186283    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d02cc4-93c2-45c3-a357-a8e0ab494395-tls-certs\") pod \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\" (UID: \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\") "
Sep 08 17:05:32.186463 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.186314    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7d02cc4-93c2-45c3-a357-a8e0ab494395-dshm\") pod \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\" (UID: \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\") "
Sep 08 17:05:32.186463 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.186380    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpqld\" (UniqueName: \"kubernetes.io/projected/d7d02cc4-93c2-45c3-a357-a8e0ab494395-kube-api-access-dpqld\") pod \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\" (UID: \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\") "
Sep 08 17:05:32.186463 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.186458    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7d02cc4-93c2-45c3-a357-a8e0ab494395-tmp-dir\") pod \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\" (UID: \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\") "
Sep 08 17:05:32.186686 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.186486    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7d02cc4-93c2-45c3-a357-a8e0ab494395-model-cache\") pod \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\" (UID: \"d7d02cc4-93c2-45c3-a357-a8e0ab494395\") "
Sep 08 17:05:32.186686 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.186556    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d02cc4-93c2-45c3-a357-a8e0ab494395-home" (OuterVolumeSpecName: "home") pod "d7d02cc4-93c2-45c3-a357-a8e0ab494395" (UID: "d7d02cc4-93c2-45c3-a357-a8e0ab494395"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:05:32.186792 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.186757    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d02cc4-93c2-45c3-a357-a8e0ab494395-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d7d02cc4-93c2-45c3-a357-a8e0ab494395" (UID: "d7d02cc4-93c2-45c3-a357-a8e0ab494395"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:05:32.186888 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.186790    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d02cc4-93c2-45c3-a357-a8e0ab494395-model-cache" (OuterVolumeSpecName: "model-cache") pod "d7d02cc4-93c2-45c3-a357-a8e0ab494395" (UID: "d7d02cc4-93c2-45c3-a357-a8e0ab494395"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:05:32.186888 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.186816    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7d02cc4-93c2-45c3-a357-a8e0ab494395-home\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 17:05:32.188751 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.188725    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d02cc4-93c2-45c3-a357-a8e0ab494395-dshm" (OuterVolumeSpecName: "dshm") pod "d7d02cc4-93c2-45c3-a357-a8e0ab494395" (UID: "d7d02cc4-93c2-45c3-a357-a8e0ab494395"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:05:32.188895 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.188747    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d02cc4-93c2-45c3-a357-a8e0ab494395-kube-api-access-dpqld" (OuterVolumeSpecName: "kube-api-access-dpqld") pod "d7d02cc4-93c2-45c3-a357-a8e0ab494395" (UID: "d7d02cc4-93c2-45c3-a357-a8e0ab494395"). InnerVolumeSpecName "kube-api-access-dpqld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:05:32.188895 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.188754    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d02cc4-93c2-45c3-a357-a8e0ab494395-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d7d02cc4-93c2-45c3-a357-a8e0ab494395" (UID: "d7d02cc4-93c2-45c3-a357-a8e0ab494395"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 17:05:32.288101 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.288062    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d02cc4-93c2-45c3-a357-a8e0ab494395-tls-certs\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 17:05:32.288101 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.288096    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7d02cc4-93c2-45c3-a357-a8e0ab494395-dshm\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 17:05:32.288101 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.288106    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpqld\" (UniqueName: \"kubernetes.io/projected/d7d02cc4-93c2-45c3-a357-a8e0ab494395-kube-api-access-dpqld\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 17:05:32.288347 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.288116    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7d02cc4-93c2-45c3-a357-a8e0ab494395-tmp-dir\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 17:05:32.288347 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.288125    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7d02cc4-93c2-45c3-a357-a8e0ab494395-model-cache\") on node \"ip-10-0-139-51.ec2.internal\" DevicePath \"\""
Sep 08 17:05:32.345431 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.345393    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff2aca5b-210e-4a0e-af0d-17a402ba4f73" path="/var/lib/kubelet/pods/ff2aca5b-210e-4a0e-af0d-17a402ba4f73/volumes"
Sep 08 17:05:32.735227 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.735195    2557 generic.go:356] "Generic (PLEG): container finished" podID="d7d02cc4-93c2-45c3-a357-a8e0ab494395" containerID="7e5535a8c0b13c6ebce8bda842c507e7e6f02e7e2e93e2f953b0c38e7e7e8644" exitCode=0
Sep 08 17:05:32.735410 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.735235    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"
Sep 08 17:05:32.735410 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.735249    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t" event={"ID":"d7d02cc4-93c2-45c3-a357-a8e0ab494395","Type":"ContainerDied","Data":"7e5535a8c0b13c6ebce8bda842c507e7e6f02e7e2e93e2f953b0c38e7e7e8644"}
Sep 08 17:05:32.735410 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.735278    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t" event={"ID":"d7d02cc4-93c2-45c3-a357-a8e0ab494395","Type":"ContainerDied","Data":"367b7fc7e3ecc8e56ed8627313439409da3f101ad7fd36dbd71d498cde1e6fea"}
Sep 08 17:05:32.735410 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.735294    2557 scope.go:117] "RemoveContainer" containerID="7e5535a8c0b13c6ebce8bda842c507e7e6f02e7e2e93e2f953b0c38e7e7e8644"
Sep 08 17:05:32.746880 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.746860    2557 scope.go:117] "RemoveContainer" containerID="7e5535a8c0b13c6ebce8bda842c507e7e6f02e7e2e93e2f953b0c38e7e7e8644"
Sep 08 17:05:32.747178 ip-10-0-139-51 kubenswrapper[2557]: E0908 17:05:32.747159    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e5535a8c0b13c6ebce8bda842c507e7e6f02e7e2e93e2f953b0c38e7e7e8644\": container with ID starting with 7e5535a8c0b13c6ebce8bda842c507e7e6f02e7e2e93e2f953b0c38e7e7e8644 not found: ID does not exist" containerID="7e5535a8c0b13c6ebce8bda842c507e7e6f02e7e2e93e2f953b0c38e7e7e8644"
Sep 08 17:05:32.747242 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.747188    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e5535a8c0b13c6ebce8bda842c507e7e6f02e7e2e93e2f953b0c38e7e7e8644"} err="failed to get container status \"7e5535a8c0b13c6ebce8bda842c507e7e6f02e7e2e93e2f953b0c38e7e7e8644\": rpc error: code = NotFound desc = could not find container \"7e5535a8c0b13c6ebce8bda842c507e7e6f02e7e2e93e2f953b0c38e7e7e8644\": container with ID starting with 7e5535a8c0b13c6ebce8bda842c507e7e6f02e7e2e93e2f953b0c38e7e7e8644 not found: ID does not exist"
Sep 08 17:05:32.753445 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.753422    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"]
Sep 08 17:05:32.758619 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:32.758598    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-15d29298-kserve-5d789f7767-wzh9t"]
Sep 08 17:05:34.345601 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:34.345563    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7d02cc4-93c2-45c3-a357-a8e0ab494395" path="/var/lib/kubelet/pods/d7d02cc4-93c2-45c3-a357-a8e0ab494395/volumes"
Sep 08 17:05:34.731654 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:34.731616    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 17:05:40.976616 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:40.976579    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 17:05:46.949324 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:46.949285    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 17:05:53.040729 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:53.040681    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 17:05:59.850891 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:05:59.850853    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-kjxk4"]
Sep 08 17:06:08.826099 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:08.826017    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-6hf2m_5f33f305-953d-4f65-95e8-7858a3f99a9d/manager/0.log"
Sep 08 17:06:08.952970 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:08.952935    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-m8wh8_85e2985e-761b-49a5-a7e4-908881c3690b/manager/0.log"
Sep 08 17:06:09.669901 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:09.669869    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86d7bb586f-xxbqf_077f72be-6477-4467-b428-fe19839b9934/copy-manifests/0.log"
Sep 08 17:06:09.677437 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:09.677407    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86d7bb586f-xxbqf_077f72be-6477-4467-b428-fe19839b9934/manager/0.log"
Sep 08 17:06:10.288066 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:10.288040    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29814765-gmpx8_f5fdae75-6aee-47b9-9a3c-21e2368ccf32/cleanup/0.log"
Sep 08 17:06:10.399633 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:10.399603    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29814780-qh5ft_0498b1a9-1215-4adf-900d-81480986051f/cleanup/0.log"
Sep 08 17:06:11.463087 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:11.463049    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-mrrww_82f2dbf1-fb30-4c46-a95f-892a1a07aa11/manager/0.log"
Sep 08 17:06:11.593430 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:11.593390    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-nk28s_9a32e226-5012-4165-a82f-e3c8991a7dc0/manager/0.log"
Sep 08 17:06:11.710446 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:11.710415    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-9cnvv_87ca0d93-517e-42a7-b46d-eeef810fe708/kuadrant-console-plugin/0.log"
Sep 08 17:06:11.833905 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:11.833806    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-gfgpt_db6bc462-753b-4f07-8bc8-95cdadb4a611/manager/0.log"
Sep 08 17:06:11.966316 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:11.966286    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-kjxk4_706cd63f-fb56-4f60-9840-9a2a1a4a26bb/limitador/0.log"
Sep 08 17:06:12.087297 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:12.087216    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-kp4f4_97a2dece-cd3b-475c-b3a5-60e042103b7d/manager/0.log"
Sep 08 17:06:12.822602 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:12.822566    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7469867b85-9j7nx_614bb2ee-e0c3-4ddb-8034-0a2231ce4608/kube-auth-proxy/0.log"
Sep 08 17:06:13.189181 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:13.189150    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7vgjf_050c1b6a-d113-4083-adb8-b94cb80b1f5c/payload-processing/0.log"
Sep 08 17:06:14.113412 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:14.113376    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpwd84_362b4cb1-2d1e-4fdf-9659-56f0f7cba944/main/0.log"
Sep 08 17:06:16.937069 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:16.937032    2557 scope.go:117] "RemoveContainer" containerID="1469a7c2df6a943239c6af72691e1165b4429eea737859aaeaf5dcf763f00a65"
Sep 08 17:06:16.945753 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:16.945729    2557 scope.go:117] "RemoveContainer" containerID="4fcc7f636a02e01d2971fac6051f0a3467f253ba401adc8fa2bf2f0e42e26b3d"
Sep 08 17:06:20.073859 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:20.073795    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6s2hc/must-gather-bzw5n"]
Sep 08 17:06:20.074408 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:20.074218    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7d02cc4-93c2-45c3-a357-a8e0ab494395" containerName="main"
Sep 08 17:06:20.074408 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:20.074237    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7d02cc4-93c2-45c3-a357-a8e0ab494395" containerName="main"
Sep 08 17:06:20.074408 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:20.074256    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff2aca5b-210e-4a0e-af0d-17a402ba4f73" containerName="main"
Sep 08 17:06:20.074408 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:20.074265    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff2aca5b-210e-4a0e-af0d-17a402ba4f73" containerName="main"
Sep 08 17:06:20.074408 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:20.074352    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff2aca5b-210e-4a0e-af0d-17a402ba4f73" containerName="main"
Sep 08 17:06:20.074408 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:20.074362    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7d02cc4-93c2-45c3-a357-a8e0ab494395" containerName="main"
Sep 08 17:06:20.077436 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:20.077413    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6s2hc/must-gather-bzw5n"
Sep 08 17:06:20.080643 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:20.080614    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6s2hc\"/\"openshift-service-ca.crt\""
Sep 08 17:06:20.081571 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:20.081547    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6s2hc\"/\"kube-root-ca.crt\""
Sep 08 17:06:20.081708 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:20.081567    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6s2hc\"/\"default-dockercfg-gwsmc\""
Sep 08 17:06:20.090027 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:20.089986    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6s2hc/must-gather-bzw5n"]
Sep 08 17:06:20.193507 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:20.193457    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psg2m\" (UniqueName: \"kubernetes.io/projected/aa2cfcde-4485-4392-8a2d-d394ecfcd8fd-kube-api-access-psg2m\") pod \"must-gather-bzw5n\" (UID: \"aa2cfcde-4485-4392-8a2d-d394ecfcd8fd\") " pod="openshift-must-gather-6s2hc/must-gather-bzw5n"
Sep 08 17:06:20.193507 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:20.193506    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aa2cfcde-4485-4392-8a2d-d394ecfcd8fd-must-gather-output\") pod \"must-gather-bzw5n\" (UID: \"aa2cfcde-4485-4392-8a2d-d394ecfcd8fd\") " pod="openshift-must-gather-6s2hc/must-gather-bzw5n"
Sep 08 17:06:20.294929 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:20.294883    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psg2m\" (UniqueName: \"kubernetes.io/projected/aa2cfcde-4485-4392-8a2d-d394ecfcd8fd-kube-api-access-psg2m\") pod \"must-gather-bzw5n\" (UID: \"aa2cfcde-4485-4392-8a2d-d394ecfcd8fd\") " pod="openshift-must-gather-6s2hc/must-gather-bzw5n"
Sep 08 17:06:20.294929 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:20.294932    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aa2cfcde-4485-4392-8a2d-d394ecfcd8fd-must-gather-output\") pod \"must-gather-bzw5n\" (UID: \"aa2cfcde-4485-4392-8a2d-d394ecfcd8fd\") " pod="openshift-must-gather-6s2hc/must-gather-bzw5n"
Sep 08 17:06:20.295315 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:20.295297    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aa2cfcde-4485-4392-8a2d-d394ecfcd8fd-must-gather-output\") pod \"must-gather-bzw5n\" (UID: \"aa2cfcde-4485-4392-8a2d-d394ecfcd8fd\") " pod="openshift-must-gather-6s2hc/must-gather-bzw5n"
Sep 08 17:06:20.303305 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:20.303271    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psg2m\" (UniqueName: \"kubernetes.io/projected/aa2cfcde-4485-4392-8a2d-d394ecfcd8fd-kube-api-access-psg2m\") pod \"must-gather-bzw5n\" (UID: \"aa2cfcde-4485-4392-8a2d-d394ecfcd8fd\") " pod="openshift-must-gather-6s2hc/must-gather-bzw5n"
Sep 08 17:06:20.387623 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:20.387521    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6s2hc/must-gather-bzw5n"
Sep 08 17:06:20.662017 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:20.661986    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6s2hc/must-gather-bzw5n"]
Sep 08 17:06:20.664582 ip-10-0-139-51 kubenswrapper[2557]: W0908 17:06:20.664555    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa2cfcde_4485_4392_8a2d_d394ecfcd8fd.slice/crio-20e944c4f054858dbb6d1e2f0aa8abd8356925648f10cf71fe5154184a02c9d7 WatchSource:0}: Error finding container 20e944c4f054858dbb6d1e2f0aa8abd8356925648f10cf71fe5154184a02c9d7: Status 404 returned error can't find the container with id 20e944c4f054858dbb6d1e2f0aa8abd8356925648f10cf71fe5154184a02c9d7
Sep 08 17:06:20.666653 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:20.666635    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 17:06:20.981675 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:20.981578    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6s2hc/must-gather-bzw5n" event={"ID":"aa2cfcde-4485-4392-8a2d-d394ecfcd8fd","Type":"ContainerStarted","Data":"20e944c4f054858dbb6d1e2f0aa8abd8356925648f10cf71fe5154184a02c9d7"}
Sep 08 17:06:22.993183 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:22.993144    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6s2hc/must-gather-bzw5n" event={"ID":"aa2cfcde-4485-4392-8a2d-d394ecfcd8fd","Type":"ContainerStarted","Data":"78e52ed4904c78d1230585275e2196b017edcad8147c348efae92b0356f36a93"}
Sep 08 17:06:22.993563 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:22.993196    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6s2hc/must-gather-bzw5n" event={"ID":"aa2cfcde-4485-4392-8a2d-d394ecfcd8fd","Type":"ContainerStarted","Data":"9fd3dba3523e6c10661940457cd38a1f74ac579a26b00cfca2b31ea6b61dedd5"}
Sep 08 17:06:24.320732 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:24.320701    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-g67mm_0abcdd93-6ae1-4e51-a19b-911c06152822/global-pull-secret-syncer/0.log"
Sep 08 17:06:24.503504 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:24.503456    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pqhsk_bd8c514b-7ca3-43e4-b87c-8a811c04efe8/konnectivity-agent/0.log"
Sep 08 17:06:24.621935 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:24.621807    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-51.ec2.internal_555e95963eeb3a1ffd9bfc39675ae3f8/haproxy/0.log"
Sep 08 17:06:29.901509 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:29.901465    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-mrrww_82f2dbf1-fb30-4c46-a95f-892a1a07aa11/manager/0.log"
Sep 08 17:06:29.936742 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:29.936601    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-nk28s_9a32e226-5012-4165-a82f-e3c8991a7dc0/manager/0.log"
Sep 08 17:06:29.963516 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:29.963484    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-9cnvv_87ca0d93-517e-42a7-b46d-eeef810fe708/kuadrant-console-plugin/0.log"
Sep 08 17:06:30.021288 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:30.021253    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-gfgpt_db6bc462-753b-4f07-8bc8-95cdadb4a611/manager/0.log"
Sep 08 17:06:30.049045 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:30.049013    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-kjxk4_706cd63f-fb56-4f60-9840-9a2a1a4a26bb/limitador/0.log"
Sep 08 17:06:30.131398 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:30.131354    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-kp4f4_97a2dece-cd3b-475c-b3a5-60e042103b7d/manager/0.log"
Sep 08 17:06:31.964855 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:31.964812    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2wnvq_2973e6a7-3ef6-4f3e-9233-e0435d98a6e2/node-exporter/0.log"
Sep 08 17:06:31.994236 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:31.994210    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2wnvq_2973e6a7-3ef6-4f3e-9233-e0435d98a6e2/kube-rbac-proxy/0.log"
Sep 08 17:06:32.040343 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:32.040317    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2wnvq_2973e6a7-3ef6-4f3e-9233-e0435d98a6e2/init-textfile/0.log"
Sep 08 17:06:35.150615 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:35.150525    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6s2hc/must-gather-bzw5n" podStartSLOduration=13.160563148 podStartE2EDuration="15.150499242s" totalImagesPullingTime="1.989936094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:06:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:06:23.014320583 +0000 UTC m=+2167.177292032" watchObservedRunningTime="2026-09-08 17:06:35.150499242 +0000 UTC m=+2179.313470691"
Sep 08 17:06:35.152024 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:35.151981    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2"]
Sep 08 17:06:35.158076 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:35.158040    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2"
Sep 08 17:06:35.171666 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:35.171633    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2"]
Sep 08 17:06:35.231963 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:35.231675    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp97q\" (UniqueName: \"kubernetes.io/projected/7ccb8e06-f42d-4f85-8836-b8a73a04f60f-kube-api-access-lp97q\") pod \"perf-node-gather-daemonset-qwxv2\" (UID: \"7ccb8e06-f42d-4f85-8836-b8a73a04f60f\") " pod="openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2"
Sep 08 17:06:35.231963 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:35.231738    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7ccb8e06-f42d-4f85-8836-b8a73a04f60f-podres\") pod \"perf-node-gather-daemonset-qwxv2\" (UID: \"7ccb8e06-f42d-4f85-8836-b8a73a04f60f\") " pod="openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2"
Sep 08 17:06:35.231963 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:35.231774    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ccb8e06-f42d-4f85-8836-b8a73a04f60f-sys\") pod \"perf-node-gather-daemonset-qwxv2\" (UID: \"7ccb8e06-f42d-4f85-8836-b8a73a04f60f\") " pod="openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2"
Sep 08 17:06:35.231963 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:35.231798    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ccb8e06-f42d-4f85-8836-b8a73a04f60f-lib-modules\") pod \"perf-node-gather-daemonset-qwxv2\" (UID: \"7ccb8e06-f42d-4f85-8836-b8a73a04f60f\") " pod="openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2"
Sep 08 17:06:35.231963 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:35.231850    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7ccb8e06-f42d-4f85-8836-b8a73a04f60f-proc\") pod \"perf-node-gather-daemonset-qwxv2\" (UID: \"7ccb8e06-f42d-4f85-8836-b8a73a04f60f\") " pod="openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2"
Sep 08 17:06:35.332551 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:35.332508    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7ccb8e06-f42d-4f85-8836-b8a73a04f60f-proc\") pod \"perf-node-gather-daemonset-qwxv2\" (UID: \"7ccb8e06-f42d-4f85-8836-b8a73a04f60f\") " pod="openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2"
Sep 08 17:06:35.332760 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:35.332592    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp97q\" (UniqueName: \"kubernetes.io/projected/7ccb8e06-f42d-4f85-8836-b8a73a04f60f-kube-api-access-lp97q\") pod \"perf-node-gather-daemonset-qwxv2\" (UID: \"7ccb8e06-f42d-4f85-8836-b8a73a04f60f\") " pod="openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2"
Sep 08 17:06:35.332760 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:35.332616    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7ccb8e06-f42d-4f85-8836-b8a73a04f60f-podres\") pod \"perf-node-gather-daemonset-qwxv2\" (UID: \"7ccb8e06-f42d-4f85-8836-b8a73a04f60f\") " pod="openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2"
Sep 08 17:06:35.332760 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:35.332639    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ccb8e06-f42d-4f85-8836-b8a73a04f60f-sys\") pod \"perf-node-gather-daemonset-qwxv2\" (UID: \"7ccb8e06-f42d-4f85-8836-b8a73a04f60f\") " pod="openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2"
Sep 08 17:06:35.332760 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:35.332655    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ccb8e06-f42d-4f85-8836-b8a73a04f60f-lib-modules\") pod \"perf-node-gather-daemonset-qwxv2\" (UID: \"7ccb8e06-f42d-4f85-8836-b8a73a04f60f\") " pod="openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2"
Sep 08 17:06:35.332760 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:35.332663    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7ccb8e06-f42d-4f85-8836-b8a73a04f60f-proc\") pod \"perf-node-gather-daemonset-qwxv2\" (UID: \"7ccb8e06-f42d-4f85-8836-b8a73a04f60f\") " pod="openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2"
Sep 08 17:06:35.332760 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:35.332752    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ccb8e06-f42d-4f85-8836-b8a73a04f60f-sys\") pod \"perf-node-gather-daemonset-qwxv2\" (UID: \"7ccb8e06-f42d-4f85-8836-b8a73a04f60f\") " pod="openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2"
Sep 08 17:06:35.333054 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:35.332794    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ccb8e06-f42d-4f85-8836-b8a73a04f60f-lib-modules\") pod \"perf-node-gather-daemonset-qwxv2\" (UID: \"7ccb8e06-f42d-4f85-8836-b8a73a04f60f\") " pod="openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2"
Sep 08 17:06:35.333054 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:35.332792    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7ccb8e06-f42d-4f85-8836-b8a73a04f60f-podres\") pod \"perf-node-gather-daemonset-qwxv2\" (UID: \"7ccb8e06-f42d-4f85-8836-b8a73a04f60f\") " pod="openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2"
Sep 08 17:06:35.348852 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:35.348783    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp97q\" (UniqueName: \"kubernetes.io/projected/7ccb8e06-f42d-4f85-8836-b8a73a04f60f-kube-api-access-lp97q\") pod \"perf-node-gather-daemonset-qwxv2\" (UID: \"7ccb8e06-f42d-4f85-8836-b8a73a04f60f\") " pod="openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2"
Sep 08 17:06:35.477069 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:35.477029    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2"
Sep 08 17:06:35.768193 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:35.768150    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2"]
Sep 08 17:06:36.085321 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:36.085228    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2" event={"ID":"7ccb8e06-f42d-4f85-8836-b8a73a04f60f","Type":"ContainerStarted","Data":"cf54fc99b402a5fca04ceecd68e258b902ea3a7a4a6149f7cc67607ab7dfeb0b"}
Sep 08 17:06:36.085321 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:36.085293    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2" event={"ID":"7ccb8e06-f42d-4f85-8836-b8a73a04f60f","Type":"ContainerStarted","Data":"08226684afbe8e7e98f8abce3d3baf601bef59071875cfca29e6387df6160a06"}
Sep 08 17:06:36.085557 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:36.085335    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2"
Sep 08 17:06:36.107906 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:36.107823    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2" podStartSLOduration=1.107800617 podStartE2EDuration="1.107800617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 17:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:06:36.10696597 +0000 UTC m=+2180.269937560" watchObservedRunningTime="2026-09-08 17:06:36.107800617 +0000 UTC m=+2180.270772067"
Sep 08 17:06:37.289805 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:37.289775    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kbwhw_195311dd-1be6-4f3d-ba2f-c7cdc374e97d/dns/0.log"
Sep 08 17:06:37.310280 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:37.310244    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kbwhw_195311dd-1be6-4f3d-ba2f-c7cdc374e97d/kube-rbac-proxy/0.log"
Sep 08 17:06:37.359945 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:37.359915    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rj64h_ad2a7c50-5f92-449d-a650-ed26da4359b7/dns-node-resolver/0.log"
Sep 08 17:06:37.962143 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:37.962106    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7967f76576-9zbkf_d92d5e13-8f2c-4850-a003-248b524e9b6b/registry/0.log"
Sep 08 17:06:38.014349 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:38.014315    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m4n88_afe27658-50c9-47a8-884a-8ddc05f375ed/node-ca/0.log"
Sep 08 17:06:39.299346 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:39.299311    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7469867b85-9j7nx_614bb2ee-e0c3-4ddb-8034-0a2231ce4608/kube-auth-proxy/0.log"
Sep 08 17:06:39.424089 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:39.424058    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7vgjf_050c1b6a-d113-4083-adb8-b94cb80b1f5c/payload-processing/0.log"
Sep 08 17:06:40.381458 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:40.381424    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-shbt6_3ac1828f-7510-4cfe-ad8f-19cc7f57d3df/serve-healthcheck-canary/0.log"
Sep 08 17:06:40.917985 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:40.917950    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7qclj_478b2574-9f73-4791-8411-90135387272a/kube-rbac-proxy/0.log"
Sep 08 17:06:40.975713 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:40.975674    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7qclj_478b2574-9f73-4791-8411-90135387272a/exporter/0.log"
Sep 08 17:06:40.999188 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:40.999156    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7qclj_478b2574-9f73-4791-8411-90135387272a/extractor/0.log"
Sep 08 17:06:42.108485 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:42.108450    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6s2hc/perf-node-gather-daemonset-qwxv2"
Sep 08 17:06:43.402657 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:43.402623    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-6hf2m_5f33f305-953d-4f65-95e8-7858a3f99a9d/manager/0.log"
Sep 08 17:06:43.726381 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:43.726343    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-m8wh8_85e2985e-761b-49a5-a7e4-908881c3690b/manager/0.log"
Sep 08 17:06:44.051969 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:44.051878    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86d7bb586f-xxbqf_077f72be-6477-4467-b428-fe19839b9934/manager/0.log"
Sep 08 17:06:44.077177 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:44.077148    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86d7bb586f-xxbqf_077f72be-6477-4467-b428-fe19839b9934/copy-manifests/0.log"
Sep 08 17:06:46.462918 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:46.462888    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7469867b85-9j7nx_614bb2ee-e0c3-4ddb-8034-0a2231ce4608/kube-auth-proxy/0.log"
Sep 08 17:06:46.562520 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:46.562484    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7vgjf_050c1b6a-d113-4083-adb8-b94cb80b1f5c/payload-processing/0.log"
Sep 08 17:06:48.353425 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:48.353387    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7469867b85-9j7nx_614bb2ee-e0c3-4ddb-8034-0a2231ce4608/kube-auth-proxy/0.log"
Sep 08 17:06:48.447388 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:48.447346    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7vgjf_050c1b6a-d113-4083-adb8-b94cb80b1f5c/payload-processing/0.log"
Sep 08 17:06:50.963322 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:50.963288    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-55s6v_0fb47bbb-ca12-44e0-b61e-76a3196c05e6/migrator/0.log"
Sep 08 17:06:50.985477 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:50.985443    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-55s6v_0fb47bbb-ca12-44e0-b61e-76a3196c05e6/graceful-termination/0.log"
Sep 08 17:06:52.672941 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:52.672909    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-62zlz_10038ceb-d586-4fb2-99e2-dc326d9daa8c/kube-multus-additional-cni-plugins/0.log"
Sep 08 17:06:52.701901 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:52.701869    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-62zlz_10038ceb-d586-4fb2-99e2-dc326d9daa8c/egress-router-binary-copy/0.log"
Sep 08 17:06:52.723787 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:52.723752    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-62zlz_10038ceb-d586-4fb2-99e2-dc326d9daa8c/cni-plugins/0.log"
Sep 08 17:06:52.743451 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:52.743420    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-62zlz_10038ceb-d586-4fb2-99e2-dc326d9daa8c/bond-cni-plugin/0.log"
Sep 08 17:06:52.764087 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:52.764056    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-62zlz_10038ceb-d586-4fb2-99e2-dc326d9daa8c/routeoverride-cni/0.log"
Sep 08 17:06:52.785107 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:52.785070    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-62zlz_10038ceb-d586-4fb2-99e2-dc326d9daa8c/whereabouts-cni-bincopy/0.log"
Sep 08 17:06:52.806857 ip-10-0-139-51 kubenswrapper[2557]: I0908 17:06:52.806810    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-62zlz_10038ceb-d586-4fb2-99e2-dc326d9daa8c/whereabouts-cni/0.log"