Sep 11 18:08:14.454016 ip-10-0-134-111 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 18:08:14.454030 ip-10-0-134-111 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 18:08:14.454040 ip-10-0-134-111 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 18:08:14.454494 ip-10-0-134-111 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 18:08:25.041591 ip-10-0-134-111 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 11 18:08:25.041606 ip-10-0-134-111 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ee16ef13de914925b3d071968686116b --
Sep 11 18:10:01.289877 ip-10-0-134-111 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 18:10:01.717713 ip-10-0-134-111 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 18:10:01.717713 ip-10-0-134-111 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 18:10:01.717713 ip-10-0-134-111 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 18:10:01.717713 ip-10-0-134-111 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 18:10:01.717713 ip-10-0-134-111 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 18:10:01.718372 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.717757    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 18:10:01.721177 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721160    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:10:01.721177 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721176    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:10:01.721247 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721180    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:10:01.721247 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721183    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:10:01.721247 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721186    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:10:01.721247 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721189    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:10:01.721247 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721192    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:10:01.721247 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721195    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:10:01.721247 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721198    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:10:01.721247 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721200    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:10:01.721247 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721203    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:10:01.721247 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721206    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:10:01.721247 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721208    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:10:01.721247 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721211    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:10:01.721247 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721214    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:10:01.721247 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721217    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:10:01.721247 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721219    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:10:01.721247 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721222    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:10:01.721247 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721224    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:10:01.721247 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721231    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:10:01.721247 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721235    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:10:01.721247 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721237    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:10:01.721729 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721240    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:10:01.721729 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721243    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:10:01.721729 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721245    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:10:01.721729 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721248    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:10:01.721729 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721252    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:10:01.721729 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721254    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:10:01.721729 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721257    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:10:01.721729 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721260    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:10:01.721729 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721263    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:10:01.721729 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721265    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:10:01.721729 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721268    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:10:01.721729 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721271    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:10:01.721729 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721273    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:10:01.721729 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721276    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:10:01.721729 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721279    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:10:01.721729 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721282    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:10:01.721729 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721284    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:10:01.721729 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721287    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:10:01.721729 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721289    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:10:01.721729 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721292    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:10:01.722252 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721294    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:10:01.722252 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721298    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:10:01.722252 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721302    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:10:01.722252 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721305    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:10:01.722252 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721307    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:10:01.722252 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721310    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:10:01.722252 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721312    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:10:01.722252 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721315    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:10:01.722252 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721317    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:10:01.722252 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721320    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:10:01.722252 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721323    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:10:01.722252 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721325    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:10:01.722252 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721328    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:10:01.722252 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721331    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:10:01.722252 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721334    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:10:01.722252 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721336    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:10:01.722252 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721339    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:10:01.722252 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721343    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:10:01.722252 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721346    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:10:01.722741 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721348    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:10:01.722741 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721351    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:10:01.722741 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721353    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:10:01.722741 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721356    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:10:01.722741 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721359    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:10:01.722741 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721361    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:10:01.722741 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721364    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:10:01.722741 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721366    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:10:01.722741 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721369    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:10:01.722741 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721372    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:10:01.722741 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721374    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:10:01.722741 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721377    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:10:01.722741 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721379    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:10:01.722741 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721382    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:10:01.722741 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721384    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:10:01.722741 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721389    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:10:01.722741 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721392    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:10:01.722741 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721395    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:10:01.722741 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721400    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:10:01.723192 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721403    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:10:01.723192 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721406    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:10:01.723192 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721409    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:10:01.723192 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721411    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:10:01.723192 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721414    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:10:01.723192 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721417    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:10:01.723192 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721419    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:10:01.723192 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721422    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:10:01.723192 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721425    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:10:01.723192 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721428    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:10:01.723192 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721430    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:10:01.723192 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721433    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:10:01.723192 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721436    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:10:01.723192 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721438    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:10:01.723192 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721441    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:10:01.723192 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721443    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:10:01.723192 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721446    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:10:01.723192 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721448    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:10:01.723192 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.721451    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:10:01.723666 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723100    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:10:01.723666 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723107    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:10:01.723666 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723110    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:10:01.723666 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723113    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:10:01.723666 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723116    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:10:01.723666 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723120    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:10:01.723666 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723122    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:10:01.723666 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723125    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:10:01.723666 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723128    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:10:01.723666 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723130    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:10:01.723666 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723133    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:10:01.723666 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723136    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:10:01.723666 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723138    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:10:01.723666 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723141    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:10:01.723666 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723143    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:10:01.723666 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723147    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:10:01.723666 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723151    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:10:01.723666 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723154    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:10:01.723666 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723157    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:10:01.724133 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723161    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:10:01.724133 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723165    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:10:01.724133 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723168    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:10:01.724133 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723171    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:10:01.724133 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723175    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:10:01.724133 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723178    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:10:01.724133 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723180    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:10:01.724133 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723183    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:10:01.724133 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723186    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:10:01.724133 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723189    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:10:01.724133 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723191    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:10:01.724133 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723195    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:10:01.724133 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723199    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:10:01.724133 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723202    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:10:01.724133 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723205    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:10:01.724133 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723208    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:10:01.724133 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723211    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:10:01.724133 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723213    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:10:01.724133 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723216    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:10:01.724652 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723219    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:10:01.724652 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723221    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:10:01.724652 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723224    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:10:01.724652 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723226    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:10:01.724652 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723229    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:10:01.724652 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723231    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:10:01.724652 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723234    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:10:01.724652 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723237    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:10:01.724652 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723239    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:10:01.724652 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723242    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:10:01.724652 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723245    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:10:01.724652 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723247    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:10:01.724652 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723250    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:10:01.724652 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723253    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:10:01.724652 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723257    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:10:01.724652 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723260    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:10:01.724652 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723263    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:10:01.724652 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723267    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:10:01.724652 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723270    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:10:01.724652 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723272    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:10:01.725143 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723275    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:10:01.725143 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723278    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:10:01.725143 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723280    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:10:01.725143 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723283    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:10:01.725143 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723285    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:10:01.725143 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723288    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:10:01.725143 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723291    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:10:01.725143 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723293    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:10:01.725143 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723296    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:10:01.725143 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723298    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:10:01.725143 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723301    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:10:01.725143 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723303    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:10:01.725143 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723306    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:10:01.725143 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723309    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:10:01.725143 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723311    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:10:01.725143 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723314    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:10:01.725143 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723316    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:10:01.725143 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723321    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:10:01.725143 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723325    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:10:01.725603 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723328    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:10:01.725603 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723330    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:10:01.725603 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723333    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:10:01.725603 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723335    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:10:01.725603 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723338    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:10:01.725603 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723340    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:10:01.725603 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723344    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:10:01.725603 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723347    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:10:01.725603 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723350    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:10:01.725603 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723353    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:10:01.725603 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723355    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:10:01.725603 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723358    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:10:01.725603 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723360    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:10:01.725603 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723363    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:10:01.725603 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723365    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:10:01.725603 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723368    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:10:01.725603 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723371    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:10:01.725603 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723373    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:10:01.725603 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723376    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:10:01.725603 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723378    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:10:01.726092 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723381    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:10:01.726092 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723383    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:10:01.726092 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723824    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:10:01.726092 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723832    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:10:01.726092 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723836    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:10:01.726092 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723839    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:10:01.726092 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723842    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:10:01.726092 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723844    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:10:01.726092 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723847    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:10:01.726092 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723850    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:10:01.726092 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723853    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:10:01.726092 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723855    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:10:01.726092 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723858    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:10:01.726092 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723860    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:10:01.726092 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723863    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:10:01.726092 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723867    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:10:01.726092 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723870    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:10:01.726092 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723873    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:10:01.726531 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723876    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:10:01.726531 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723879    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:10:01.726531 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723886    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:10:01.726531 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723889    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:10:01.726531 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723892    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:10:01.726531 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723895    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:10:01.726531 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723898    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:10:01.726531 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723901    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:10:01.726531 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723903    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:10:01.726531 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723906    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:10:01.726531 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723909    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:10:01.726531 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723912    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:10:01.726531 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723915    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:10:01.726531 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723917    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:10:01.726531 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723921    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:10:01.726531 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723924    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:10:01.726531 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723927    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:10:01.726531 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723930    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:10:01.726531 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723933    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:10:01.727013 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723935    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:10:01.727013 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723938    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:10:01.727013 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723940    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:10:01.727013 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723943    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:10:01.727013 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723946    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:10:01.727013 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723948    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:10:01.727013 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723951    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:10:01.727013 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723953    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:10:01.727013 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723956    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:10:01.727013 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723958    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:10:01.727013 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723961    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:10:01.727013 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723963    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:10:01.727013 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723966    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:10:01.727013 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723968    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:10:01.727013 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723970    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:10:01.727013 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723974    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:10:01.727013 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723982    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:10:01.727013 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723985    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:10:01.727013 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723987    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:10:01.727013 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723989    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:10:01.727501 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723992    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:10:01.727501 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723995    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:10:01.727501 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.723997    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:10:01.727501 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724000    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:10:01.727501 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724002    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:10:01.727501 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724005    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:10:01.727501 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724007    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:10:01.727501 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724009    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:10:01.727501 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724012    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:10:01.727501 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724014    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:10:01.727501 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724017    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:10:01.727501 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724020    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:10:01.727501 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724023    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:10:01.727501 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724026    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:10:01.727501 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724029    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:10:01.727501 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724031    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:10:01.727501 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724033    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:10:01.727501 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724036    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:10:01.727501 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724039    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:10:01.727501 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724041    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:10:01.728003 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724044    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:10:01.728003 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724046    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:10:01.728003 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724049    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:10:01.728003 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724051    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:10:01.728003 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724054    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:10:01.728003 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724056    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:10:01.728003 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724058    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:10:01.728003 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724061    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:10:01.728003 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724063    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:10:01.728003 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724071    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:10:01.728003 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724074    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:10:01.728003 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724076    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:10:01.728003 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724079    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:10:01.728003 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724081    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:10:01.728003 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724084    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:10:01.728003 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724086    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:10:01.728003 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724089    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:10:01.728003 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724091    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:10:01.728003 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724093    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:10:01.728003 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724096    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:10:01.728485 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724098    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:10:01.728485 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724101    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:10:01.728485 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724104    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:10:01.728485 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724106    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:10:01.728485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724171    2576 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 18:10:01.728485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724208    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 18:10:01.728485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724217    2576 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 18:10:01.728485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724222    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 18:10:01.728485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724226    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 18:10:01.728485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724230    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 18:10:01.728485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724234    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 18:10:01.728485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724238    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 18:10:01.728485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724241    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 18:10:01.728485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724244    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 18:10:01.728485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724247    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 18:10:01.728485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724251    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 18:10:01.728485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724254    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 18:10:01.728485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724257    2576 flags.go:64] FLAG: --cgroup-root=""
Sep 11 18:10:01.728485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724259    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 18:10:01.728485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724263    2576 flags.go:64] FLAG: --client-ca-file=""
Sep 11 18:10:01.728485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724266    2576 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 18:10:01.728485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724269    2576 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 18:10:01.728485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724276    2576 flags.go:64] FLAG: --cluster-domain=""
Sep 11 18:10:01.728485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724279    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 18:10:01.729066 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724282    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 18:10:01.729066 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724285    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 18:10:01.729066 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724289    2576 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 18:10:01.729066 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724293    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 18:10:01.729066 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724296    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 18:10:01.729066 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724299    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 18:10:01.729066 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724302    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 18:10:01.729066 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724305    2576 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 18:10:01.729066 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724310    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 18:10:01.729066 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724313    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 18:10:01.729066 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724316    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 18:10:01.729066 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724319    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 18:10:01.729066 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724323    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 18:10:01.729066 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724326    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 18:10:01.729066 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724329    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 18:10:01.729066 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724332    2576 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 18:10:01.729066 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724335    2576 flags.go:64] FLAG: --enable-server="true"
Sep 11 18:10:01.729066 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724338    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 18:10:01.729066 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724343    2576 flags.go:64] FLAG: --event-burst="100"
Sep 11 18:10:01.729066 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724346    2576 flags.go:64] FLAG: --event-qps="50"
Sep 11 18:10:01.729066 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724349    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 18:10:01.729066 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724352    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 18:10:01.729066 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724354    2576 flags.go:64] FLAG: --eviction-hard=""
Sep 11 18:10:01.729066 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724358    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 18:10:01.729729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724361    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 18:10:01.729729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724364    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 18:10:01.729729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724367    2576 flags.go:64] FLAG: --eviction-soft=""
Sep 11 18:10:01.729729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724369    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 18:10:01.729729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724372    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 18:10:01.729729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724375    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 18:10:01.729729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724378    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 18:10:01.729729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724381    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 18:10:01.729729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724384    2576 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 18:10:01.729729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724387    2576 flags.go:64] FLAG: --feature-gates=""
Sep 11 18:10:01.729729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724391    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 18:10:01.729729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724394    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 18:10:01.729729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724397    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 18:10:01.729729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724400    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 18:10:01.729729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724402    2576 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 18:10:01.729729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724405    2576 flags.go:64] FLAG: --help="false"
Sep 11 18:10:01.729729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724409    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-134-111.ec2.internal"
Sep 11 18:10:01.729729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724413    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 18:10:01.729729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724416    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 18:10:01.729729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724419    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 18:10:01.729729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724422    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 18:10:01.729729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724425    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 18:10:01.729729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724428    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 18:10:01.729729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724431    2576 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724434    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724437    2576 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724440    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724443    2576 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724445    2576 flags.go:64] FLAG: --kube-reserved=""
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724448    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724451    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724454    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724457    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724460    2576 flags.go:64] FLAG: --lock-file=""
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724462    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724465    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724468    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724473    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724476    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724479    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724482    2576 flags.go:64] FLAG: --logging-format="text"
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724485    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724488    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724491    2576 flags.go:64] FLAG: --manifest-url=""
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724494    2576 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724498    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724501    2576 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724504    2576 flags.go:64] FLAG: --max-pods="110"
Sep 11 18:10:01.730380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724509    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 18:10:01.731029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724512    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 18:10:01.731029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724515    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 18:10:01.731029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724517    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 18:10:01.731029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724520    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 18:10:01.731029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724523    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 18:10:01.731029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724526    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 18:10:01.731029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724533    2576 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 18:10:01.731029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724536    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 18:10:01.731029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724539    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 18:10:01.731029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724542    2576 flags.go:64] FLAG: --pod-cidr=""
Sep 11 18:10:01.731029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724545    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 18:10:01.731029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724550    2576 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 18:10:01.731029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724553    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 18:10:01.731029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724555    2576 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 18:10:01.731029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724558    2576 flags.go:64] FLAG: --port="10250"
Sep 11 18:10:01.731029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724561    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 18:10:01.731029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724564    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-086eb77aea7b26479"
Sep 11 18:10:01.731029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724567    2576 flags.go:64] FLAG: --qos-reserved=""
Sep 11 18:10:01.731029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724570    2576 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 18:10:01.731029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724573    2576 flags.go:64] FLAG: --register-node="true"
Sep 11 18:10:01.731029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724576    2576 flags.go:64] FLAG: --register-with-taints=""
Sep 11 18:10:01.731029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724593    2576 flags.go:64] FLAG: --registry-burst="10"
Sep 11 18:10:01.731029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724596    2576 flags.go:64] FLAG: --registry-qps="5"
Sep 11 18:10:01.731029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724599    2576 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724603    2576 flags.go:64] FLAG: --reserved-memory=""
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724606    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724609    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724612    2576 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724615    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724618    2576 flags.go:64] FLAG: --runonce="false"
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724621    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724625    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724628    2576 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724631    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724633    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724636    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724639    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724642    2576 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724645    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724648    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724651    2576 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724655    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724658    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724661    2576 flags.go:64] FLAG: --system-cgroups=""
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724664    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724669    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724672    2576 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724675    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 18:10:01.731827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724679    2576 flags.go:64] FLAG: --tls-min-version=""
Sep 11 18:10:01.732692 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724682    2576 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 18:10:01.732692 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724685    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 18:10:01.732692 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724688    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 18:10:01.732692 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724690    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 18:10:01.732692 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724693    2576 flags.go:64] FLAG: --v="2"
Sep 11 18:10:01.732692 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724697    2576 flags.go:64] FLAG: --version="false"
Sep 11 18:10:01.732692 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724701    2576 flags.go:64] FLAG: --vmodule=""
Sep 11 18:10:01.732692 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724705    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 18:10:01.732692 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.724708    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 18:10:01.732692 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724877    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:10:01.732692 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724881    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:10:01.732692 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724885    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:10:01.732692 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724889    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:10:01.732692 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724892    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:10:01.732692 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724896    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:10:01.732692 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724899    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:10:01.732692 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724902    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:10:01.732692 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724905    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:10:01.732692 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724908    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:10:01.732692 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724911    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:10:01.732692 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724914    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:10:01.733206 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724917    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:10:01.733206 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724921    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:10:01.733206 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724924    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:10:01.733206 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724942    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:10:01.733206 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724946    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:10:01.733206 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724950    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:10:01.733206 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724953    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:10:01.733206 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724955    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:10:01.733206 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724958    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:10:01.733206 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724960    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:10:01.733206 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724963    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:10:01.733206 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724966    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:10:01.733206 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724968    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:10:01.733206 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724971    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:10:01.733206 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724973    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:10:01.733206 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724976    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:10:01.733206 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724979    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:10:01.733206 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724988    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:10:01.733206 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724991    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:10:01.733742 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.724994    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:10:01.733742 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725005    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:10:01.733742 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725008    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:10:01.733742 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725010    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:10:01.733742 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725013    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:10:01.733742 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725015    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:10:01.733742 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725019    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:10:01.733742 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725022    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:10:01.733742 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725025    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:10:01.733742 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725027    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:10:01.733742 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725030    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:10:01.733742 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725033    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:10:01.733742 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725036    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:10:01.733742 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725038    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:10:01.733742 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725042    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:10:01.733742 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725045    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:10:01.733742 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725047    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:10:01.733742 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725050    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:10:01.733742 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725052    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:10:01.733742 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725055    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:10:01.734253 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725058    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:10:01.734253 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725060    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:10:01.734253 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725063    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:10:01.734253 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725066    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:10:01.734253 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725068    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:10:01.734253 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725071    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:10:01.734253 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725074    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:10:01.734253 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725078    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:10:01.734253 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725081    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:10:01.734253 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725084    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:10:01.734253 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725086    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:10:01.734253 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725089    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:10:01.734253 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725091    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:10:01.734253 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725094    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:10:01.734253 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725102    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:10:01.734253 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725106    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:10:01.734253 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725110    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:10:01.734253 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725121    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:10:01.734253 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725125    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:10:01.734737 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725128    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:10:01.734737 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725131    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:10:01.734737 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725134    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:10:01.734737 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725137    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:10:01.734737 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725140    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:10:01.734737 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725142    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:10:01.734737 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725145    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:10:01.734737 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725149    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:10:01.734737 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725152    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:10:01.734737 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725155    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:10:01.734737 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725157    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:10:01.734737 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725160    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:10:01.734737 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725163    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:10:01.734737 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725166    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:10:01.734737 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725169    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:10:01.734737 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725171    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:10:01.734737 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725174    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:10:01.734737 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725177    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:10:01.734737 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725180    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:10:01.734737 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725182    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:10:01.735227 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725185    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:10:01.735227 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725188    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:10:01.735227 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725190    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:10:01.735227 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725193    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:10:01.735227 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725195    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:10:01.735227 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725198    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:10:01.735227 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725201    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:10:01.735227 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725203    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:10:01.735227 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.725211    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:10:01.735227 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.725216    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 18:10:01.735227 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.734140    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 18:10:01.735227 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.734157    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 18:10:01.735227 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.734170    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 18:10:01.735227 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.734177    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 18:10:01.735227 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734232    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:10:01.735609 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734237    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:10:01.735609 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734240    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:10:01.735609 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734244    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:10:01.735609 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734247    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:10:01.735609 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734250    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:10:01.735609 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734253    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:10:01.735609 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734257    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:10:01.735609 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734260    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:10:01.735609 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734265    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:10:01.735609 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734269    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:10:01.735609 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734273    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:10:01.735609 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734276    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:10:01.735609 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734280    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:10:01.735609 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734284    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:10:01.735609 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734288    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:10:01.735609 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734291    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:10:01.735609 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734294    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:10:01.735609 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734297    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:10:01.735609 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734299    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:10:01.736063 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734302    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:10:01.736063 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734305    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:10:01.736063 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734307    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:10:01.736063 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734310    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:10:01.736063 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734313    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:10:01.736063 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734316    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:10:01.736063 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734319    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:10:01.736063 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734321    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:10:01.736063 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734324    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:10:01.736063 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734327    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:10:01.736063 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734330    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:10:01.736063 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734332    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:10:01.736063 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734335    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:10:01.736063 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734338    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:10:01.736063 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734341    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:10:01.736063 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734343    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:10:01.736063 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734346    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:10:01.736063 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734349    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:10:01.736063 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734361    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:10:01.736517 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734365    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:10:01.736517 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734368    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:10:01.736517 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734372    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:10:01.736517 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734374    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:10:01.736517 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734377    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:10:01.736517 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734380    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:10:01.736517 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734383    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:10:01.736517 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734387    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:10:01.736517 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734389    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:10:01.736517 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734392    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:10:01.736517 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734395    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:10:01.736517 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734397    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:10:01.736517 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734400    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:10:01.736517 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734403    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:10:01.736517 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734406    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:10:01.736517 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734409    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:10:01.736517 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734411    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:10:01.736517 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734414    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:10:01.736517 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734417    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:10:01.736517 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734419    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:10:01.737069 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734422    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:10:01.737069 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734425    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:10:01.737069 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734428    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:10:01.737069 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734431    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:10:01.737069 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734433    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:10:01.737069 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734436    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:10:01.737069 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734439    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:10:01.737069 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734441    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:10:01.737069 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734459    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:10:01.737069 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734463    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:10:01.737069 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734466    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:10:01.737069 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734469    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:10:01.737069 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734472    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:10:01.737069 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734475    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:10:01.737069 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734478    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:10:01.737069 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734480    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:10:01.737069 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734483    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:10:01.737069 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734486    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:10:01.737069 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734489    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:10:01.737069 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734492    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:10:01.737542 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734495    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:10:01.737542 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734497    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:10:01.737542 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734500    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:10:01.737542 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734502    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:10:01.737542 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734505    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:10:01.737542 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734507    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:10:01.737542 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734510    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:10:01.737542 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734513    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:10:01.737542 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734515    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:10:01.737542 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734518    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:10:01.737542 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734521    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:10:01.737542 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734524    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:10:01.737542 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734526    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:10:01.737542 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734530    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:10:01.737542 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734533    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:10:01.737542 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734535    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:10:01.737542 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734538    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:10:01.737542 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734541    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:10:01.737542 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734543    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:10:01.737542 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734546    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:10:01.738079 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.734551    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 18:10:01.738079 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734679    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:10:01.738079 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734685    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:10:01.738079 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734688    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:10:01.738079 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734691    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:10:01.738079 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734694    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:10:01.738079 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734697    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:10:01.738079 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734700    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:10:01.738079 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734703    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:10:01.738079 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734705    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:10:01.738079 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734708    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:10:01.738079 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734711    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:10:01.738079 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734714    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:10:01.738079 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734718    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:10:01.738079 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734721    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:10:01.738447 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734724    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:10:01.738447 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734727    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:10:01.738447 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734729    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:10:01.738447 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734732    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:10:01.738447 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734735    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:10:01.738447 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734737    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:10:01.738447 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734740    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:10:01.738447 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734743    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:10:01.738447 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734746    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:10:01.738447 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734749    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:10:01.738447 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734752    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:10:01.738447 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734755    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:10:01.738447 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734758    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:10:01.738447 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734761    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:10:01.738447 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734763    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:10:01.738447 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734766    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:10:01.738447 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734768    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:10:01.738447 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734771    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:10:01.738447 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734774    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:10:01.738447 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734776    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:10:01.739025 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734779    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:10:01.739025 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734782    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:10:01.739025 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734784    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:10:01.739025 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734787    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:10:01.739025 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734789    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:10:01.739025 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734792    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:10:01.739025 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734795    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:10:01.739025 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734798    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:10:01.739025 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734802    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:10:01.739025 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734805    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:10:01.739025 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734808    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:10:01.739025 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734812    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:10:01.739025 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734815    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:10:01.739025 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734818    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:10:01.739025 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734821    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:10:01.739025 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734824    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:10:01.739025 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734827    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:10:01.739025 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734829    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:10:01.739025 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734832    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:10:01.739475 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734834    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:10:01.739475 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734837    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:10:01.739475 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734840    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:10:01.739475 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734843    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:10:01.739475 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734845    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:10:01.739475 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734848    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:10:01.739475 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734850    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:10:01.739475 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734853    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:10:01.739475 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734855    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:10:01.739475 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734858    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:10:01.739475 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734861    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:10:01.739475 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734864    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:10:01.739475 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734868    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:10:01.739475 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734871    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:10:01.739475 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734873    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:10:01.739475 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734876    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:10:01.739475 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734878    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:10:01.739475 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734881    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:10:01.739475 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734884    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:10:01.739956 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734886    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:10:01.739956 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734889    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:10:01.739956 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734891    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:10:01.739956 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734894    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:10:01.739956 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734896    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:10:01.739956 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734899    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:10:01.739956 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734902    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:10:01.739956 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734904    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:10:01.739956 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734907    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:10:01.739956 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734909    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:10:01.739956 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734912    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:10:01.739956 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734914    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:10:01.739956 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734917    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:10:01.739956 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734920    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:10:01.739956 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734922    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:10:01.739956 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734925    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:10:01.739956 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734927    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:10:01.739956 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734930    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:10:01.739956 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734932    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:10:01.739956 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734935    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:10:01.740448 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734937    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:10:01.740448 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734940    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:10:01.740448 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734942    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:10:01.740448 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734945    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:10:01.740448 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734947    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:10:01.740448 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734950    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:10:01.740448 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:01.734952    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:10:01.740448 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.734957    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 18:10:01.740448 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.735057    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 18:10:01.740448 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.739202    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 18:10:01.740448 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.740248    2576 server.go:1019] "Starting client certificate rotation"
Sep 11 18:10:01.740448 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.740359    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 18:10:01.740448 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.740388    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 18:10:01.743048 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.743035    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 18:10:01.746337 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.746322    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 18:10:01.757176 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.757155    2576 log.go:25] "Validated CRI v1 runtime API"
Sep 11 18:10:01.760485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.760470    2576 log.go:25] "Validated CRI v1 image API"
Sep 11 18:10:01.761349 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.761333    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 18:10:01.767601 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.767473    2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ab22e3ce-6619-47e8-bfc9-ac48db3507fd:/dev/nvme0n1p3 b652ead9-8340-4291-91f5-90505c7b429e:/dev/nvme0n1p4]
Sep 11 18:10:01.767601 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.767552    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 18:10:01.773805 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.773688    2576 manager.go:216] Machine: {Timestamp:2026-09-11 18:10:01.77226972 +0000 UTC m=+0.373630874 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099795 MemoryCapacity:33164464128 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2777b8eeb36505ce791af53a98caf1 SystemUUID:ec2777b8-eeb3-6505-ce79-1af53a98caf1 BootID:ee16ef13-de91-4925-b3d0-71968686116b Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582230016 Type:vfs Inodes:4048396 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:05:7e:e1:cf:e5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:05:7e:e1:cf:e5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:2a:ac:3d:c1:40 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164464128 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 18:10:01.773805 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.773794    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 18:10:01.773932 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.773877    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 18:10:01.777308 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.777286    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 18:10:01.777448 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.777312    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-111.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 18:10:01.778229 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.778219    2576 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 18:10:01.778267 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.778231    2576 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 18:10:01.778267 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.778239    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 18:10:01.778267 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.778252    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 18:10:01.779451 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.779431    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 18:10:01.779532 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.779522    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 18:10:01.784054 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.784043    2576 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 18:10:01.784162 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.784153    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 18:10:01.784197 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.784181    2576 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 18:10:01.784197 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.784190    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 18:10:01.784257 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.784201    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 18:10:01.784257 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.784210    2576 kubelet.go:398] "Adding apiserver pod source"
Sep 11 18:10:01.784257 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.784221    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 18:10:01.784573 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.784560    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 18:10:01.784630 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.784596    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 18:10:01.787687 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.787670    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 18:10:01.790600 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.790573    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 18:10:01.790651 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.790617    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 18:10:01.792762 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.792749    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 18:10:01.792794 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.792771    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 18:10:01.792794 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.792782    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 18:10:01.792794 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.792791    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 18:10:01.792877 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.792804    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 18:10:01.792877 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.792812    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 18:10:01.792877 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.792832    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 18:10:01.792877 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.792841    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 18:10:01.792877 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.792850    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 18:10:01.792877 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.792858    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 18:10:01.793030 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.792880    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 18:10:01.793030 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.792890    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 18:10:01.793866 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.793852    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 18:10:01.793866 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.793866    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 18:10:01.797670 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:01.797648    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 18:10:01.797754 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.797679    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-111.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 18:10:01.797754 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.797734    2576 server.go:1268] "Started kubelet"
Sep 11 18:10:01.797865 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.797800    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 18:10:01.797912 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.797844    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 18:10:01.797955 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.797915    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 18:10:01.798225 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:01.798204    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 18:10:01.798536 ip-10-0-134-111 systemd[1]: Started Kubernetes Kubelet.
Sep 11 18:10:01.798993 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.798980    2576 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 18:10:01.799572 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.799553    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 18:10:01.804553 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:01.804529    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 18:10:01.806160 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.806146    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 18:10:01.806222 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.806158    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 18:10:01.806271 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:01.806249    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-111.ec2.internal\" not found"
Sep 11 18:10:01.806913 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.806899    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 18:10:01.806988 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.806900    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 18:10:01.806988 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.806935    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 18:10:01.806988 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.806982    2576 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 18:10:01.806988 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.806988    2576 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 18:10:01.812366 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.812344    2576 factory.go:55] Registering systemd factory
Sep 11 18:10:01.812366 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.812367    2576 factory.go:223] Registration of the systemd container factory successfully
Sep 11 18:10:01.812704 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:01.812680    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 18:10:01.812704 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:01.812684    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 18:10:01.812954 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.812936    2576 factory.go:153] Registering CRI-O factory
Sep 11 18:10:01.812988 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.812960    2576 factory.go:223] Registration of the crio container factory successfully
Sep 11 18:10:01.813028 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.813022    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 18:10:01.813064 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.813041    2576 factory.go:103] Registering Raw factory
Sep 11 18:10:01.813064 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.813058    2576 manager.go:1196] Started watching for new ooms in manager
Sep 11 18:10:01.813479 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.813467    2576 manager.go:349] Starting recovery of all containers
Sep 11 18:10:01.813981 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:01.812793    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-111.ec2.internal.18d456a113f892d2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-111.ec2.internal,UID:ip-10-0-134-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-111.ec2.internal,},FirstTimestamp:2026-09-11 18:10:01.797710546 +0000 UTC m=+0.399071701,LastTimestamp:2026-09-11 18:10:01.797710546 +0000 UTC m=+0.399071701,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-111.ec2.internal,}"
Sep 11 18:10:01.823508 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.823351    2576 manager.go:354] Recovery completed
Sep 11 18:10:01.827839 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.827827    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:10:01.830525 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.830497    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:10:01.830613 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.830537    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:10:01.830613 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.830550    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:10:01.831120 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.831103    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 18:10:01.831120 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.831117    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 18:10:01.831226 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.831132    2576 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 18:10:01.831226 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.831153    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:10:01.831931 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.831917    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:10:01.831986 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.831946    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:10:01.831986 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.831960    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:10:01.832825 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:01.832759    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-111.ec2.internal.18d456a115ed4475  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-111.ec2.internal,UID:ip-10-0-134-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-111.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-111.ec2.internal,},FirstTimestamp:2026-09-11 18:10:01.830524021 +0000 UTC m=+0.431885174,LastTimestamp:2026-09-11 18:10:01.830524021 +0000 UTC m=+0.431885174,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-111.ec2.internal,}"
Sep 11 18:10:01.833255 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.833244    2576 policy_none.go:49] "None policy: Start"
Sep 11 18:10:01.833294 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.833259    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 18:10:01.833294 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.833269    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 18:10:01.835163 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.835151    2576 policy_none.go:47] "Start"
Sep 11 18:10:01.843340 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:01.843249    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-111.ec2.internal.18d456a115ed9259  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-111.ec2.internal,UID:ip-10-0-134-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-111.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-111.ec2.internal,},FirstTimestamp:2026-09-11 18:10:01.830543961 +0000 UTC m=+0.431905115,LastTimestamp:2026-09-11 18:10:01.830543961 +0000 UTC m=+0.431905115,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-111.ec2.internal,}"
Sep 11 18:10:01.857119 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:01.857037    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-111.ec2.internal.18d456a115edb837  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-111.ec2.internal,UID:ip-10-0-134-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-111.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-111.ec2.internal,},FirstTimestamp:2026-09-11 18:10:01.830553655 +0000 UTC m=+0.431914810,LastTimestamp:2026-09-11 18:10:01.830553655 +0000 UTC m=+0.431914810,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-111.ec2.internal,}"
Sep 11 18:10:01.880939 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.863251    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sjqhh"
Sep 11 18:10:01.880939 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:01.867677    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-111.ec2.internal.18d456a115ed4475\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-111.ec2.internal.18d456a115ed4475  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-111.ec2.internal,UID:ip-10-0-134-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-111.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-111.ec2.internal,},FirstTimestamp:2026-09-11 18:10:01.830524021 +0000 UTC m=+0.431885174,LastTimestamp:2026-09-11 18:10:01.831930747 +0000 UTC m=+0.433291902,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-111.ec2.internal,}"
Sep 11 18:10:01.880939 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.874635    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sjqhh"
Sep 11 18:10:01.880939 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.874950    2576 manager.go:337] "Starting Device Plugin manager"
Sep 11 18:10:01.880939 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:01.874997    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 18:10:01.880939 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.875010    2576 server.go:87] "Starting device plugin registration server"
Sep 11 18:10:01.880939 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.875419    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 18:10:01.880939 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.875433    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 18:10:01.880939 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.875693    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 18:10:01.880939 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.875769    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 18:10:01.880939 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.875778    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 18:10:01.880939 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:01.876321    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 18:10:01.880939 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:01.876373    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-111.ec2.internal\" not found"
Sep 11 18:10:01.902669 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.902637    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 18:10:01.903905 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.903888    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 18:10:01.903957 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.903946    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 18:10:01.903992 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.903973    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 18:10:01.904035 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:01.904014    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 18:10:01.906534 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.906513    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:10:01.975699 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.975568    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:10:01.976697 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.976679    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:10:01.976769 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.976711    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:10:01.976769 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.976721    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:10:01.976769 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.976742    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-111.ec2.internal"
Sep 11 18:10:01.985918 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:01.985898    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-111.ec2.internal"
Sep 11 18:10:01.986013 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:01.985922    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-111.ec2.internal\": node \"ip-10-0-134-111.ec2.internal\" not found"
Sep 11 18:10:02.004640 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.004617    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-111.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-111.ec2.internal"]
Sep 11 18:10:02.004726 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.004680    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:10:02.006046 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.006030    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:10:02.006122 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.006059    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:10:02.006122 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.006069    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:10:02.008258 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.008238    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8471b73601d05635d4bf15db7b2236da-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-111.ec2.internal\" (UID: \"8471b73601d05635d4bf15db7b2236da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-111.ec2.internal"
Sep 11 18:10:02.008258 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.008253    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:10:02.008376 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.008266    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8471b73601d05635d4bf15db7b2236da-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-111.ec2.internal\" (UID: \"8471b73601d05635d4bf15db7b2236da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-111.ec2.internal"
Sep 11 18:10:02.008428 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.008381    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-111.ec2.internal"
Sep 11 18:10:02.008428 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.008418    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:10:02.009010 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.008994    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:10:02.009094 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.009025    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:10:02.009094 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.009035    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:10:02.009094 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.008999    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:10:02.009094 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.009087    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:10:02.009209 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.009104    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:10:02.011295 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.011277    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-111.ec2.internal"
Sep 11 18:10:02.011395 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.011303    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:10:02.012042 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.012027    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:10:02.012125 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.012052    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:10:02.012125 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.012062    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:10:02.032838 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:02.032816    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-111.ec2.internal\" not found"
Sep 11 18:10:02.035237 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:02.035222    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-111.ec2.internal\" not found" node="ip-10-0-134-111.ec2.internal"
Sep 11 18:10:02.039649 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:02.039629    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-111.ec2.internal\" not found" node="ip-10-0-134-111.ec2.internal"
Sep 11 18:10:02.108625 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.108578    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8471b73601d05635d4bf15db7b2236da-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-111.ec2.internal\" (UID: \"8471b73601d05635d4bf15db7b2236da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-111.ec2.internal"
Sep 11 18:10:02.108625 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.108622    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8471b73601d05635d4bf15db7b2236da-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-111.ec2.internal\" (UID: \"8471b73601d05635d4bf15db7b2236da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-111.ec2.internal"
Sep 11 18:10:02.108804 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.108642    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6fbb6fad8f6855b4beb91cec165f915-config\") pod \"kube-apiserver-proxy-ip-10-0-134-111.ec2.internal\" (UID: \"d6fbb6fad8f6855b4beb91cec165f915\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-111.ec2.internal"
Sep 11 18:10:02.108804 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.108711    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8471b73601d05635d4bf15db7b2236da-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-111.ec2.internal\" (UID: \"8471b73601d05635d4bf15db7b2236da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-111.ec2.internal"
Sep 11 18:10:02.108804 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.108711    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8471b73601d05635d4bf15db7b2236da-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-111.ec2.internal\" (UID: \"8471b73601d05635d4bf15db7b2236da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-111.ec2.internal"
Sep 11 18:10:02.133180 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:02.133156    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-111.ec2.internal\" not found"
Sep 11 18:10:02.208797 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.208754    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6fbb6fad8f6855b4beb91cec165f915-config\") pod \"kube-apiserver-proxy-ip-10-0-134-111.ec2.internal\" (UID: \"d6fbb6fad8f6855b4beb91cec165f915\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-111.ec2.internal"
Sep 11 18:10:02.208925 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.208821    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6fbb6fad8f6855b4beb91cec165f915-config\") pod \"kube-apiserver-proxy-ip-10-0-134-111.ec2.internal\" (UID: \"d6fbb6fad8f6855b4beb91cec165f915\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-111.ec2.internal"
Sep 11 18:10:02.233933 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:02.233878    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-111.ec2.internal\" not found"
Sep 11 18:10:02.334673 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:02.334629    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-111.ec2.internal\" not found"
Sep 11 18:10:02.337779 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.337763    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-111.ec2.internal"
Sep 11 18:10:02.341860 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.341836    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-111.ec2.internal"
Sep 11 18:10:02.436263 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:02.436213    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-111.ec2.internal\" not found"
Sep 11 18:10:02.537101 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:02.537030    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-111.ec2.internal\" not found"
Sep 11 18:10:02.637293 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:02.637247    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-111.ec2.internal\" not found"
Sep 11 18:10:02.737848 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:02.737815    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-111.ec2.internal\" not found"
Sep 11 18:10:02.740018 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.739993    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 18:10:02.740158 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.740141    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 18:10:02.838362 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:02.838293    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-111.ec2.internal\" not found"
Sep 11 18:10:02.880216 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.880167    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 18:05:02 +0000 UTC" deadline="2028-05-13 05:42:57.808530573 +0000 UTC"
Sep 11 18:10:02.880344 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.880217    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14627h32m54.928318317s"
Sep 11 18:10:02.939254 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:02.939220    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-111.ec2.internal\" not found"
Sep 11 18:10:02.962127 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.962095    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:10:02.963979 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.963962    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:10:02.964060 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.963996    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:10:02.964060 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:02.964011    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:10:03.010857 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.010824    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:10:03.025414 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.025392    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:10:03.030708 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:03.030654    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8471b73601d05635d4bf15db7b2236da.slice/crio-dab730856e7c20525584a4c7848614ec563c99750eede53d35702af0903e61f4 WatchSource:0}: Error finding container dab730856e7c20525584a4c7848614ec563c99750eede53d35702af0903e61f4: Status 404 returned error can't find the container with id dab730856e7c20525584a4c7848614ec563c99750eede53d35702af0903e61f4
Sep 11 18:10:03.030936 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:03.030919    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6fbb6fad8f6855b4beb91cec165f915.slice/crio-a87233e76833d8d3ae6002244d2191011f7f052f295be2d6dbb93656a527d8d5 WatchSource:0}: Error finding container a87233e76833d8d3ae6002244d2191011f7f052f295be2d6dbb93656a527d8d5: Status 404 returned error can't find the container with id a87233e76833d8d3ae6002244d2191011f7f052f295be2d6dbb93656a527d8d5
Sep 11 18:10:03.034732 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.034716    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:10:03.110332 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.110227    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-111.ec2.internal"
Sep 11 18:10:03.131274 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.131246    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 18:10:03.132307 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.132287    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-111.ec2.internal"
Sep 11 18:10:03.143691 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.143668    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 18:10:03.281518 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.281488    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:10:03.786778 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.786741    2576 apiserver.go:52] "Watching apiserver"
Sep 11 18:10:03.797158 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.797125    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 18:10:03.799251 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.799186    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-j7jrb","kube-system/kube-apiserver-proxy-ip-10-0-134-111.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx","openshift-cluster-node-tuning-operator/tuned-r4w8t","openshift-dns/node-resolver-nqz9g","openshift-image-registry/node-ca-ckcv2","openshift-multus/multus-jkms9","openshift-multus/network-metrics-daemon-44nnd","kube-system/global-pull-secret-syncer-db9kz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-111.ec2.internal","openshift-multus/multus-additional-cni-plugins-759db","openshift-network-diagnostics/network-check-target-qhvvw","openshift-network-operator/iptables-alerter-7967x","openshift-ovn-kubernetes/ovnkube-node-rxclq"]
Sep 11 18:10:03.801823 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.801786    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.804247 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.804216    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7967x"
Sep 11 18:10:03.804831 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.804812    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 18:10:03.804939 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.804843    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 18:10:03.804999 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.804953    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fmtn9\""
Sep 11 18:10:03.805091 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.805073    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 18:10:03.805150 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.805095    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 18:10:03.806427 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.806386    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 18:10:03.807636 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.806887    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx"
Sep 11 18:10:03.807636 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.807046    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:10:03.807636 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.807052    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-w5vnr\""
Sep 11 18:10:03.807636 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.807424    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 18:10:03.807636 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.807441    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 18:10:03.809182 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.808954    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.809800 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.809770    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 18:10:03.810698 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.810675    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 18:10:03.810827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.810805    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nqxnw\""
Sep 11 18:10:03.811037 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.811019    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 18:10:03.811420 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.811398    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nqz9g"
Sep 11 18:10:03.812317 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.812297    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 18:10:03.812422 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.812355    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nbp5d\""
Sep 11 18:10:03.812541 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.812518    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:10:03.814004 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.813963    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ckcv2"
Sep 11 18:10:03.814108 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.814023    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 18:10:03.814568 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.814546    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mh8lm\""
Sep 11 18:10:03.814685 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.814626    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 18:10:03.816596 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.816558    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 18:10:03.817038 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.817017    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gbqvz\""
Sep 11 18:10:03.817291 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.817270    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:03.817364 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.817289    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 18:10:03.817422 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:03.817359    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db9kz" podUID="99a9405b-bada-4d50-a3cb-3cf9eb701eee"
Sep 11 18:10:03.817880 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.817859    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 18:10:03.819645 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.819540    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j7jrb"
Sep 11 18:10:03.820111 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820004    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-multus-conf-dir\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.820111 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820044    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-host-run-multus-certs\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.820268 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820129    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-etc-sysconfig\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.820268 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820160    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-etc-sysctl-conf\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.820268 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820192    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01a91968-3ccc-4258-b71e-f98563bf0138-cni-binary-copy\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.820268 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820228    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-etc-sysctl-d\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.820486 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820269    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-etc-systemd\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.820486 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820299    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-multus-socket-dir-parent\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.820486 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820323    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-host-run-netns\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.820486 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820345    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-hostroot\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.820486 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820370    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4fef7440-cfca-4eaf-a0f9-c102ea506ab8-iptables-alerter-script\") pod \"iptables-alerter-7967x\" (UID: \"4fef7440-cfca-4eaf-a0f9-c102ea506ab8\") " pod="openshift-network-operator/iptables-alerter-7967x"
Sep 11 18:10:03.820486 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820397    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fef7440-cfca-4eaf-a0f9-c102ea506ab8-host-slash\") pod \"iptables-alerter-7967x\" (UID: \"4fef7440-cfca-4eaf-a0f9-c102ea506ab8\") " pod="openshift-network-operator/iptables-alerter-7967x"
Sep 11 18:10:03.820486 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820422    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-var-lib-kubelet\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.820486 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-system-cni-dir\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.820486 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820468    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-multus-cni-dir\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.820957 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820492    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/01a91968-3ccc-4258-b71e-f98563bf0138-multus-daemon-config\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.820957 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820517    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-etc-kubernetes\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.820957 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820553    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63b887df-53bd-4441-a220-b5d82b09bbbf-etc-tuned\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.820957 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820576    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-cnibin\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.820957 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820630    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-host-var-lib-cni-bin\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.820957 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820666    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-host-var-lib-cni-multus\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.820957 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820694    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc85a151-5b62-439d-98e1-8a117c5a2f3b-sys-fs\") pod \"aws-ebs-csi-driver-node-jx8qx\" (UID: \"dc85a151-5b62-439d-98e1-8a117c5a2f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx"
Sep 11 18:10:03.820957 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820718    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-etc-modprobe-d\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.820957 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820743    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-sys\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.820957 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820767    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-host-run-k8s-cni-cncf-io\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.820957 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820791    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc85a151-5b62-439d-98e1-8a117c5a2f3b-registration-dir\") pod \"aws-ebs-csi-driver-node-jx8qx\" (UID: \"dc85a151-5b62-439d-98e1-8a117c5a2f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx"
Sep 11 18:10:03.820957 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820832    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc85a151-5b62-439d-98e1-8a117c5a2f3b-device-dir\") pod \"aws-ebs-csi-driver-node-jx8qx\" (UID: \"dc85a151-5b62-439d-98e1-8a117c5a2f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx"
Sep 11 18:10:03.820957 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820889    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc85a151-5b62-439d-98e1-8a117c5a2f3b-etc-selinux\") pod \"aws-ebs-csi-driver-node-jx8qx\" (UID: \"dc85a151-5b62-439d-98e1-8a117c5a2f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx"
Sep 11 18:10:03.820957 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820917    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-789b7\" (UniqueName: \"kubernetes.io/projected/dc85a151-5b62-439d-98e1-8a117c5a2f3b-kube-api-access-789b7\") pod \"aws-ebs-csi-driver-node-jx8qx\" (UID: \"dc85a151-5b62-439d-98e1-8a117c5a2f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx"
Sep 11 18:10:03.820957 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820946    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8ndx\" (UniqueName: \"kubernetes.io/projected/63b887df-53bd-4441-a220-b5d82b09bbbf-kube-api-access-v8ndx\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.821795 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.820977    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-host-var-lib-kubelet\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.821795 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.821005    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjcks\" (UniqueName: \"kubernetes.io/projected/01a91968-3ccc-4258-b71e-f98563bf0138-kube-api-access-pjcks\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.821795 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.821047    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkwvt\" (UniqueName: \"kubernetes.io/projected/4fef7440-cfca-4eaf-a0f9-c102ea506ab8-kube-api-access-zkwvt\") pod \"iptables-alerter-7967x\" (UID: \"4fef7440-cfca-4eaf-a0f9-c102ea506ab8\") " pod="openshift-network-operator/iptables-alerter-7967x"
Sep 11 18:10:03.821795 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.821071    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc85a151-5b62-439d-98e1-8a117c5a2f3b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jx8qx\" (UID: \"dc85a151-5b62-439d-98e1-8a117c5a2f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx"
Sep 11 18:10:03.821795 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.821108    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc85a151-5b62-439d-98e1-8a117c5a2f3b-socket-dir\") pod \"aws-ebs-csi-driver-node-jx8qx\" (UID: \"dc85a151-5b62-439d-98e1-8a117c5a2f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx"
Sep 11 18:10:03.821795 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.821138    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-lib-modules\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.821795 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.821162    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-host\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.821795 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.821185    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-etc-kubernetes\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.821795 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.821208    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-run\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.821795 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.821230    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63b887df-53bd-4441-a220-b5d82b09bbbf-tmp\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.821795 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.821660    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 18:10:03.822350 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.821854    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 18:10:03.822772 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.822753    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:03.823454 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.823432    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zjdn5\""
Sep 11 18:10:03.823582 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.823506    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 18:10:03.825916 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.825897    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:03.826009 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:03.825967    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44nnd" podUID="6609afb5-5000-42c3-814e-9f9ee86a32d4"
Sep 11 18:10:03.826716 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.826358    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 18:10:03.826716 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.826546    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 18:10:03.826716 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.826582    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 18:10:03.826716 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.826643    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 18:10:03.827023 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.826888    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7k9mq\""
Sep 11 18:10:03.827106 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.827083    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 18:10:03.827160 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.827114    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 18:10:03.828256 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.828237    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-759db"
Sep 11 18:10:03.830469 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.830444    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 18:10:03.830469 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.830468    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9bt7x\""
Sep 11 18:10:03.830824 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.830806    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 18:10:03.831158 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.831134    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:03.831232 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:03.831198    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvvw" podUID="c882403c-4f90-4d58-a269-9b897f0ba2e8"
Sep 11 18:10:03.855662 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.855413    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-n9xzn"
Sep 11 18:10:03.865468 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.865221    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-n9xzn"
Sep 11 18:10:03.908242 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.908212    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 18:10:03.910357 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.910276    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-111.ec2.internal" event={"ID":"8471b73601d05635d4bf15db7b2236da","Type":"ContainerStarted","Data":"dab730856e7c20525584a4c7848614ec563c99750eede53d35702af0903e61f4"}
Sep 11 18:10:03.911927 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.911897    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-111.ec2.internal" event={"ID":"d6fbb6fad8f6855b4beb91cec165f915","Type":"ContainerStarted","Data":"a87233e76833d8d3ae6002244d2191011f7f052f295be2d6dbb93656a527d8d5"}
Sep 11 18:10:03.921710 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.921437    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fef7440-cfca-4eaf-a0f9-c102ea506ab8-host-slash\") pod \"iptables-alerter-7967x\" (UID: \"4fef7440-cfca-4eaf-a0f9-c102ea506ab8\") " pod="openshift-network-operator/iptables-alerter-7967x"
Sep 11 18:10:03.921710 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.921543    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-multus-cni-dir\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.921710 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.921545    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fef7440-cfca-4eaf-a0f9-c102ea506ab8-host-slash\") pod \"iptables-alerter-7967x\" (UID: \"4fef7440-cfca-4eaf-a0f9-c102ea506ab8\") " pod="openshift-network-operator/iptables-alerter-7967x"
Sep 11 18:10:03.921710 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.921584    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/01a91968-3ccc-4258-b71e-f98563bf0138-multus-daemon-config\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.921710 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.921629    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-host-var-lib-cni-multus\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.921710 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.921638    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-multus-cni-dir\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.921710 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.921659    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99a9405b-bada-4d50-a3cb-3cf9eb701eee-kubelet-config\") pod \"global-pull-secret-syncer-db9kz\" (UID: \"99a9405b-bada-4d50-a3cb-3cf9eb701eee\") " pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:03.921710 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.921685    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-host-run-k8s-cni-cncf-io\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.921710 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.921709    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc85a151-5b62-439d-98e1-8a117c5a2f3b-device-dir\") pod \"aws-ebs-csi-driver-node-jx8qx\" (UID: \"dc85a151-5b62-439d-98e1-8a117c5a2f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx"
Sep 11 18:10:03.922255 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.921863    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc85a151-5b62-439d-98e1-8a117c5a2f3b-device-dir\") pod \"aws-ebs-csi-driver-node-jx8qx\" (UID: \"dc85a151-5b62-439d-98e1-8a117c5a2f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx"
Sep 11 18:10:03.922255 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.921905    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-host-var-lib-cni-multus\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.922255 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.921942    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-host-run-k8s-cni-cncf-io\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.922255 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.921975    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc85a151-5b62-439d-98e1-8a117c5a2f3b-etc-selinux\") pod \"aws-ebs-csi-driver-node-jx8qx\" (UID: \"dc85a151-5b62-439d-98e1-8a117c5a2f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx"
Sep 11 18:10:03.922255 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.922013    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-789b7\" (UniqueName: \"kubernetes.io/projected/dc85a151-5b62-439d-98e1-8a117c5a2f3b-kube-api-access-789b7\") pod \"aws-ebs-csi-driver-node-jx8qx\" (UID: \"dc85a151-5b62-439d-98e1-8a117c5a2f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx"
Sep 11 18:10:03.922255 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.922045    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8ndx\" (UniqueName: \"kubernetes.io/projected/63b887df-53bd-4441-a220-b5d82b09bbbf-kube-api-access-v8ndx\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.922255 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.922098    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0a9600b-c5a4-4e01-99b5-fe0598a4cb52-tmp-dir\") pod \"node-resolver-nqz9g\" (UID: \"e0a9600b-c5a4-4e01-99b5-fe0598a4cb52\") " pod="openshift-dns/node-resolver-nqz9g"
Sep 11 18:10:03.922255 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.922125    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-systemd-units\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:03.922255 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.922152    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjcks\" (UniqueName: \"kubernetes.io/projected/01a91968-3ccc-4258-b71e-f98563bf0138-kube-api-access-pjcks\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.922255 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.922182    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc85a151-5b62-439d-98e1-8a117c5a2f3b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jx8qx\" (UID: \"dc85a151-5b62-439d-98e1-8a117c5a2f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx"
Sep 11 18:10:03.922955 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.922314    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/01a91968-3ccc-4258-b71e-f98563bf0138-multus-daemon-config\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.922955 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.922445    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc85a151-5b62-439d-98e1-8a117c5a2f3b-etc-selinux\") pod \"aws-ebs-csi-driver-node-jx8qx\" (UID: \"dc85a151-5b62-439d-98e1-8a117c5a2f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx"
Sep 11 18:10:03.922955 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.922486    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc85a151-5b62-439d-98e1-8a117c5a2f3b-socket-dir\") pod \"aws-ebs-csi-driver-node-jx8qx\" (UID: \"dc85a151-5b62-439d-98e1-8a117c5a2f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx"
Sep 11 18:10:03.922955 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.922514    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-lib-modules\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.922955 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.922548    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99a9405b-bada-4d50-a3cb-3cf9eb701eee-original-pull-secret\") pod \"global-pull-secret-syncer-db9kz\" (UID: \"99a9405b-bada-4d50-a3cb-3cf9eb701eee\") " pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:03.922955 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.922577    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-var-lib-openvswitch\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:03.922955 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.922620    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs\") pod \"network-metrics-daemon-44nnd\" (UID: \"6609afb5-5000-42c3-814e-9f9ee86a32d4\") " pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:03.922955 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.922685    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63b887df-53bd-4441-a220-b5d82b09bbbf-tmp\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.922955 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.922711    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e0a9600b-c5a4-4e01-99b5-fe0598a4cb52-hosts-file\") pod \"node-resolver-nqz9g\" (UID: \"e0a9600b-c5a4-4e01-99b5-fe0598a4cb52\") " pod="openshift-dns/node-resolver-nqz9g"
Sep 11 18:10:03.922955 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.922735    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232-agent-certs\") pod \"konnectivity-agent-j7jrb\" (UID: \"f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232\") " pod="kube-system/konnectivity-agent-j7jrb"
Sep 11 18:10:03.922955 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.922761    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-host-kubelet\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:03.922955 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.922786    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-log-socket\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:03.922955 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.922814    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-multus-conf-dir\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.922955 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.922824    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc85a151-5b62-439d-98e1-8a117c5a2f3b-socket-dir\") pod \"aws-ebs-csi-driver-node-jx8qx\" (UID: \"dc85a151-5b62-439d-98e1-8a117c5a2f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx"
Sep 11 18:10:03.922955 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.922840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-etc-sysctl-conf\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.922955 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.922933    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc85a151-5b62-439d-98e1-8a117c5a2f3b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jx8qx\" (UID: \"dc85a151-5b62-439d-98e1-8a117c5a2f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx"
Sep 11 18:10:03.923832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.923075    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-etc-sysctl-conf\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.923832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.923118    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxnjj\" (UniqueName: \"kubernetes.io/projected/c882403c-4f90-4d58-a269-9b897f0ba2e8-kube-api-access-nxnjj\") pod \"network-check-target-qhvvw\" (UID: \"c882403c-4f90-4d58-a269-9b897f0ba2e8\") " pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:03.923832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.923147    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c70edc6-7469-4d7c-b3f1-5d88caac76aa-serviceca\") pod \"node-ca-ckcv2\" (UID: \"9c70edc6-7469-4d7c-b3f1-5d88caac76aa\") " pod="openshift-image-registry/node-ca-ckcv2"
Sep 11 18:10:03.923832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.923168    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-multus-conf-dir\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.923832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.923176    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-host-slash\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:03.923832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.923379    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-lib-modules\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.923832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.923439    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-etc-openvswitch\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:03.923832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.923474    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-host-run-ovn-kubernetes\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:03.923832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.923503    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:03.923832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.923562    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-etc-sysctl-d\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.923832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.923591    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe7af037-cf3e-4ae8-bdf0-f458a249950f-system-cni-dir\") pod \"multus-additional-cni-plugins-759db\" (UID: \"fe7af037-cf3e-4ae8-bdf0-f458a249950f\") " pod="openshift-multus/multus-additional-cni-plugins-759db"
Sep 11 18:10:03.923832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.923597    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 18:10:03.923832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.923620    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-host-cni-bin\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:03.923832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.923687    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb6146c9-7f2c-43a4-8e60-3e995a078646-ovn-node-metrics-cert\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:03.923832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.923750    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktm2t\" (UniqueName: \"kubernetes.io/projected/6609afb5-5000-42c3-814e-9f9ee86a32d4-kube-api-access-ktm2t\") pod \"network-metrics-daemon-44nnd\" (UID: \"6609afb5-5000-42c3-814e-9f9ee86a32d4\") " pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:03.923832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.923802    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-multus-socket-dir-parent\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.923832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.923815    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-etc-sysctl-d\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.924711 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.923851    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-host-run-netns\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.924711 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.923858    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-multus-socket-dir-parent\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.924711 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.923895    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-hostroot\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.924711 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.923932    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-host-run-netns\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.924711 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.923987    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4fef7440-cfca-4eaf-a0f9-c102ea506ab8-iptables-alerter-script\") pod \"iptables-alerter-7967x\" (UID: \"4fef7440-cfca-4eaf-a0f9-c102ea506ab8\") " pod="openshift-network-operator/iptables-alerter-7967x"
Sep 11 18:10:03.924711 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924015    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-var-lib-kubelet\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.924711 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924047    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe7af037-cf3e-4ae8-bdf0-f458a249950f-cnibin\") pod \"multus-additional-cni-plugins-759db\" (UID: \"fe7af037-cf3e-4ae8-bdf0-f458a249950f\") " pod="openshift-multus/multus-additional-cni-plugins-759db"
Sep 11 18:10:03.924711 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924095    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwgbq\" (UniqueName: \"kubernetes.io/projected/cb6146c9-7f2c-43a4-8e60-3e995a078646-kube-api-access-nwgbq\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:03.924711 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924105    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-var-lib-kubelet\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.924711 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924136    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-hostroot\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.924711 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924179    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-system-cni-dir\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.924711 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924212    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-etc-kubernetes\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.924711 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924246    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63b887df-53bd-4441-a220-b5d82b09bbbf-etc-tuned\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.924711 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924270    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-system-cni-dir\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.924711 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924273    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6wnq\" (UniqueName: \"kubernetes.io/projected/9c70edc6-7469-4d7c-b3f1-5d88caac76aa-kube-api-access-f6wnq\") pod \"node-ca-ckcv2\" (UID: \"9c70edc6-7469-4d7c-b3f1-5d88caac76aa\") " pod="openshift-image-registry/node-ca-ckcv2"
Sep 11 18:10:03.924711 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924310    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99a9405b-bada-4d50-a3cb-3cf9eb701eee-dbus\") pod \"global-pull-secret-syncer-db9kz\" (UID: \"99a9405b-bada-4d50-a3cb-3cf9eb701eee\") " pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:03.924711 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924329    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-etc-kubernetes\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.925636 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924332    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232-konnectivity-ca\") pod \"konnectivity-agent-j7jrb\" (UID: \"f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232\") " pod="kube-system/konnectivity-agent-j7jrb"
Sep 11 18:10:03.925636 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924365    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-host-run-netns\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:03.925636 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924387    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cb6146c9-7f2c-43a4-8e60-3e995a078646-ovnkube-script-lib\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:03.925636 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924411    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-cnibin\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.925636 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924431    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-host-var-lib-cni-bin\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.925636 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924453    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc85a151-5b62-439d-98e1-8a117c5a2f3b-sys-fs\") pod \"aws-ebs-csi-driver-node-jx8qx\" (UID: \"dc85a151-5b62-439d-98e1-8a117c5a2f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx"
Sep 11 18:10:03.925636 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924473    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-etc-modprobe-d\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.925636 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924496    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-sys\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.925636 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924518    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fe7af037-cf3e-4ae8-bdf0-f458a249950f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-759db\" (UID: \"fe7af037-cf3e-4ae8-bdf0-f458a249950f\") " pod="openshift-multus/multus-additional-cni-plugins-759db"
Sep 11 18:10:03.925636 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924540    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg7bv\" (UniqueName: \"kubernetes.io/projected/fe7af037-cf3e-4ae8-bdf0-f458a249950f-kube-api-access-kg7bv\") pod \"multus-additional-cni-plugins-759db\" (UID: \"fe7af037-cf3e-4ae8-bdf0-f458a249950f\") " pod="openshift-multus/multus-additional-cni-plugins-759db"
Sep 11 18:10:03.925636 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924563    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb4nf\" (UniqueName: \"kubernetes.io/projected/e0a9600b-c5a4-4e01-99b5-fe0598a4cb52-kube-api-access-fb4nf\") pod \"node-resolver-nqz9g\" (UID: \"e0a9600b-c5a4-4e01-99b5-fe0598a4cb52\") " pod="openshift-dns/node-resolver-nqz9g"
Sep 11 18:10:03.925636 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924585    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc85a151-5b62-439d-98e1-8a117c5a2f3b-registration-dir\") pod \"aws-ebs-csi-driver-node-jx8qx\" (UID: \"dc85a151-5b62-439d-98e1-8a117c5a2f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx"
Sep 11 18:10:03.925636 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924607    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe7af037-cf3e-4ae8-bdf0-f458a249950f-cni-binary-copy\") pod \"multus-additional-cni-plugins-759db\" (UID: \"fe7af037-cf3e-4ae8-bdf0-f458a249950f\") " pod="openshift-multus/multus-additional-cni-plugins-759db"
Sep 11 18:10:03.925636 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924647    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-host-cni-netd\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:03.925636 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924672    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cb6146c9-7f2c-43a4-8e60-3e995a078646-ovnkube-config\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:03.925636 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924695    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-host-var-lib-kubelet\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.925636 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924717    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkwvt\" (UniqueName: \"kubernetes.io/projected/4fef7440-cfca-4eaf-a0f9-c102ea506ab8-kube-api-access-zkwvt\") pod \"iptables-alerter-7967x\" (UID: \"4fef7440-cfca-4eaf-a0f9-c102ea506ab8\") " pod="openshift-network-operator/iptables-alerter-7967x"
Sep 11 18:10:03.926568 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924739    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-host\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.926568 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924773    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fe7af037-cf3e-4ae8-bdf0-f458a249950f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-759db\" (UID: \"fe7af037-cf3e-4ae8-bdf0-f458a249950f\") " pod="openshift-multus/multus-additional-cni-plugins-759db"
Sep 11 18:10:03.926568 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924796    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-node-log\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:03.926568 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924818    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-etc-kubernetes\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.926568 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924841    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-run\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.926568 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924864    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe7af037-cf3e-4ae8-bdf0-f458a249950f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-759db\" (UID: \"fe7af037-cf3e-4ae8-bdf0-f458a249950f\") " pod="openshift-multus/multus-additional-cni-plugins-759db"
Sep 11 18:10:03.926568 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924890    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-host-run-multus-certs\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.926568 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924914    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-etc-sysconfig\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.926568 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924938    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c70edc6-7469-4d7c-b3f1-5d88caac76aa-host\") pod \"node-ca-ckcv2\" (UID: \"9c70edc6-7469-4d7c-b3f1-5d88caac76aa\") " pod="openshift-image-registry/node-ca-ckcv2"
Sep 11 18:10:03.926568 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924963    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-run-systemd\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:03.926568 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.924995    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-run-openvswitch\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:03.926568 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.925020    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-run-ovn\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:03.926568 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.925051    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cb6146c9-7f2c-43a4-8e60-3e995a078646-env-overrides\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:03.926568 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.925076    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01a91968-3ccc-4258-b71e-f98563bf0138-cni-binary-copy\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.926568 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.925092    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4fef7440-cfca-4eaf-a0f9-c102ea506ab8-iptables-alerter-script\") pod \"iptables-alerter-7967x\" (UID: \"4fef7440-cfca-4eaf-a0f9-c102ea506ab8\") " pod="openshift-network-operator/iptables-alerter-7967x"
Sep 11 18:10:03.926568 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.925100    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-etc-systemd\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.926568 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.925212    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc85a151-5b62-439d-98e1-8a117c5a2f3b-registration-dir\") pod \"aws-ebs-csi-driver-node-jx8qx\" (UID: \"dc85a151-5b62-439d-98e1-8a117c5a2f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx"
Sep 11 18:10:03.927480 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.925296    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-host\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.927480 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.925358    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-etc-kubernetes\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.927480 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.925479    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-etc-systemd\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.927480 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.925530    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-host-var-lib-kubelet\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.927480 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.925584    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-cnibin\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.927480 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.925619    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-host-var-lib-cni-bin\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.927480 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.925720    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc85a151-5b62-439d-98e1-8a117c5a2f3b-sys-fs\") pod \"aws-ebs-csi-driver-node-jx8qx\" (UID: \"dc85a151-5b62-439d-98e1-8a117c5a2f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx"
Sep 11 18:10:03.927480 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.925786    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-etc-modprobe-d\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.927480 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.925828    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-sys\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.927480 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.925859    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01a91968-3ccc-4258-b71e-f98563bf0138-cni-binary-copy\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.927480 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.925941    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/01a91968-3ccc-4258-b71e-f98563bf0138-host-run-multus-certs\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.927480 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.925957    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-etc-sysconfig\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.927480 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.925978    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/63b887df-53bd-4441-a220-b5d82b09bbbf-run\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.928649 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.928520    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/63b887df-53bd-4441-a220-b5d82b09bbbf-etc-tuned\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.928649 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.928563    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63b887df-53bd-4441-a220-b5d82b09bbbf-tmp\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.938507 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.938478    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjcks\" (UniqueName: \"kubernetes.io/projected/01a91968-3ccc-4258-b71e-f98563bf0138-kube-api-access-pjcks\") pod \"multus-jkms9\" (UID: \"01a91968-3ccc-4258-b71e-f98563bf0138\") " pod="openshift-multus/multus-jkms9"
Sep 11 18:10:03.940419 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.940394    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkwvt\" (UniqueName: \"kubernetes.io/projected/4fef7440-cfca-4eaf-a0f9-c102ea506ab8-kube-api-access-zkwvt\") pod \"iptables-alerter-7967x\" (UID: \"4fef7440-cfca-4eaf-a0f9-c102ea506ab8\") " pod="openshift-network-operator/iptables-alerter-7967x"
Sep 11 18:10:03.940542 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.940394    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8ndx\" (UniqueName: \"kubernetes.io/projected/63b887df-53bd-4441-a220-b5d82b09bbbf-kube-api-access-v8ndx\") pod \"tuned-r4w8t\" (UID: \"63b887df-53bd-4441-a220-b5d82b09bbbf\") " pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:03.941172 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:03.941146    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-789b7\" (UniqueName: \"kubernetes.io/projected/dc85a151-5b62-439d-98e1-8a117c5a2f3b-kube-api-access-789b7\") pod \"aws-ebs-csi-driver-node-jx8qx\" (UID: \"dc85a151-5b62-439d-98e1-8a117c5a2f3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx"
Sep 11 18:10:04.028472 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.026378    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwgbq\" (UniqueName: \"kubernetes.io/projected/cb6146c9-7f2c-43a4-8e60-3e995a078646-kube-api-access-nwgbq\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.028472 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.026429    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6wnq\" (UniqueName: \"kubernetes.io/projected/9c70edc6-7469-4d7c-b3f1-5d88caac76aa-kube-api-access-f6wnq\") pod \"node-ca-ckcv2\" (UID: \"9c70edc6-7469-4d7c-b3f1-5d88caac76aa\") " pod="openshift-image-registry/node-ca-ckcv2"
Sep 11 18:10:04.028472 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.026458    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99a9405b-bada-4d50-a3cb-3cf9eb701eee-dbus\") pod \"global-pull-secret-syncer-db9kz\" (UID: \"99a9405b-bada-4d50-a3cb-3cf9eb701eee\") " pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:04.028472 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.026484    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232-konnectivity-ca\") pod \"konnectivity-agent-j7jrb\" (UID: \"f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232\") " pod="kube-system/konnectivity-agent-j7jrb"
Sep 11 18:10:04.028472 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.026500    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-host-run-netns\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.028472 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.026525    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cb6146c9-7f2c-43a4-8e60-3e995a078646-ovnkube-script-lib\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.028472 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.026562    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fe7af037-cf3e-4ae8-bdf0-f458a249950f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-759db\" (UID: \"fe7af037-cf3e-4ae8-bdf0-f458a249950f\") " pod="openshift-multus/multus-additional-cni-plugins-759db"
Sep 11 18:10:04.028472 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.026594    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg7bv\" (UniqueName: \"kubernetes.io/projected/fe7af037-cf3e-4ae8-bdf0-f458a249950f-kube-api-access-kg7bv\") pod \"multus-additional-cni-plugins-759db\" (UID: \"fe7af037-cf3e-4ae8-bdf0-f458a249950f\") " pod="openshift-multus/multus-additional-cni-plugins-759db"
Sep 11 18:10:04.028472 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.026609    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb4nf\" (UniqueName: \"kubernetes.io/projected/e0a9600b-c5a4-4e01-99b5-fe0598a4cb52-kube-api-access-fb4nf\") pod \"node-resolver-nqz9g\" (UID: \"e0a9600b-c5a4-4e01-99b5-fe0598a4cb52\") " pod="openshift-dns/node-resolver-nqz9g"
Sep 11 18:10:04.028472 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.026639    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe7af037-cf3e-4ae8-bdf0-f458a249950f-cni-binary-copy\") pod \"multus-additional-cni-plugins-759db\" (UID: \"fe7af037-cf3e-4ae8-bdf0-f458a249950f\") " pod="openshift-multus/multus-additional-cni-plugins-759db"
Sep 11 18:10:04.028472 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.026687    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-host-cni-netd\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.028472 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.026713    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cb6146c9-7f2c-43a4-8e60-3e995a078646-ovnkube-config\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.028472 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.026741    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fe7af037-cf3e-4ae8-bdf0-f458a249950f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-759db\" (UID: \"fe7af037-cf3e-4ae8-bdf0-f458a249950f\") " pod="openshift-multus/multus-additional-cni-plugins-759db"
Sep 11 18:10:04.028472 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.026767    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-node-log\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.028472 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.026794    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe7af037-cf3e-4ae8-bdf0-f458a249950f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-759db\" (UID: \"fe7af037-cf3e-4ae8-bdf0-f458a249950f\") " pod="openshift-multus/multus-additional-cni-plugins-759db"
Sep 11 18:10:04.028472 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.026820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c70edc6-7469-4d7c-b3f1-5d88caac76aa-host\") pod \"node-ca-ckcv2\" (UID: \"9c70edc6-7469-4d7c-b3f1-5d88caac76aa\") " pod="openshift-image-registry/node-ca-ckcv2"
Sep 11 18:10:04.028472 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.026845    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-run-systemd\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.029588 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.026902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-run-openvswitch\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.029588 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.026944    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-run-ovn\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.029588 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.026971    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cb6146c9-7f2c-43a4-8e60-3e995a078646-env-overrides\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.029588 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.027004    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99a9405b-bada-4d50-a3cb-3cf9eb701eee-kubelet-config\") pod \"global-pull-secret-syncer-db9kz\" (UID: \"99a9405b-bada-4d50-a3cb-3cf9eb701eee\") " pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:04.029588 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.027040    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0a9600b-c5a4-4e01-99b5-fe0598a4cb52-tmp-dir\") pod \"node-resolver-nqz9g\" (UID: \"e0a9600b-c5a4-4e01-99b5-fe0598a4cb52\") " pod="openshift-dns/node-resolver-nqz9g"
Sep 11 18:10:04.029588 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.027068    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99a9405b-bada-4d50-a3cb-3cf9eb701eee-dbus\") pod \"global-pull-secret-syncer-db9kz\" (UID: \"99a9405b-bada-4d50-a3cb-3cf9eb701eee\") " pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:04.029588 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.027142    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-systemd-units\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.029588 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.027395    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c70edc6-7469-4d7c-b3f1-5d88caac76aa-host\") pod \"node-ca-ckcv2\" (UID: \"9c70edc6-7469-4d7c-b3f1-5d88caac76aa\") " pod="openshift-image-registry/node-ca-ckcv2"
Sep 11 18:10:04.029588 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.027440    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-host-cni-netd\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.029588 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.027474    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-node-log\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.029588 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.027666    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe7af037-cf3e-4ae8-bdf0-f458a249950f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-759db\" (UID: \"fe7af037-cf3e-4ae8-bdf0-f458a249950f\") " pod="openshift-multus/multus-additional-cni-plugins-759db"
Sep 11 18:10:04.029588 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.027797    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-run-systemd\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.029588 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.027743    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe7af037-cf3e-4ae8-bdf0-f458a249950f-cni-binary-copy\") pod \"multus-additional-cni-plugins-759db\" (UID: \"fe7af037-cf3e-4ae8-bdf0-f458a249950f\") " pod="openshift-multus/multus-additional-cni-plugins-759db"
Sep 11 18:10:04.029588 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.027866    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-run-openvswitch\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.029588 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.027954    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99a9405b-bada-4d50-a3cb-3cf9eb701eee-kubelet-config\") pod \"global-pull-secret-syncer-db9kz\" (UID: \"99a9405b-bada-4d50-a3cb-3cf9eb701eee\") " pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:04.029588 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028018    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-host-run-netns\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.029588 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.027073    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-systemd-units\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.029588 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028054    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232-konnectivity-ca\") pod \"konnectivity-agent-j7jrb\" (UID: \"f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232\") " pod="kube-system/konnectivity-agent-j7jrb"
Sep 11 18:10:04.030523 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028074    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cb6146c9-7f2c-43a4-8e60-3e995a078646-ovnkube-config\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.030523 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028080    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99a9405b-bada-4d50-a3cb-3cf9eb701eee-original-pull-secret\") pod \"global-pull-secret-syncer-db9kz\" (UID: \"99a9405b-bada-4d50-a3cb-3cf9eb701eee\") " pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:04.030523 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028112    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-var-lib-openvswitch\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.030523 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028141    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs\") pod \"network-metrics-daemon-44nnd\" (UID: \"6609afb5-5000-42c3-814e-9f9ee86a32d4\") " pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:04.030523 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028173    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e0a9600b-c5a4-4e01-99b5-fe0598a4cb52-hosts-file\") pod \"node-resolver-nqz9g\" (UID: \"e0a9600b-c5a4-4e01-99b5-fe0598a4cb52\") " pod="openshift-dns/node-resolver-nqz9g"
Sep 11 18:10:04.030523 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:04.028196    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:10:04.030523 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:04.028273    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99a9405b-bada-4d50-a3cb-3cf9eb701eee-original-pull-secret podName:99a9405b-bada-4d50-a3cb-3cf9eb701eee nodeName:}" failed. No retries permitted until 2026-09-11 18:10:04.528250294 +0000 UTC m=+3.129611443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99a9405b-bada-4d50-a3cb-3cf9eb701eee-original-pull-secret") pod "global-pull-secret-syncer-db9kz" (UID: "99a9405b-bada-4d50-a3cb-3cf9eb701eee") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:10:04.030523 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028338    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0a9600b-c5a4-4e01-99b5-fe0598a4cb52-tmp-dir\") pod \"node-resolver-nqz9g\" (UID: \"e0a9600b-c5a4-4e01-99b5-fe0598a4cb52\") " pod="openshift-dns/node-resolver-nqz9g"
Sep 11 18:10:04.030523 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028350    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cb6146c9-7f2c-43a4-8e60-3e995a078646-ovnkube-script-lib\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.030523 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028199    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232-agent-certs\") pod \"konnectivity-agent-j7jrb\" (UID: \"f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232\") " pod="kube-system/konnectivity-agent-j7jrb"
Sep 11 18:10:04.030523 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028425    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e0a9600b-c5a4-4e01-99b5-fe0598a4cb52-hosts-file\") pod \"node-resolver-nqz9g\" (UID: \"e0a9600b-c5a4-4e01-99b5-fe0598a4cb52\") " pod="openshift-dns/node-resolver-nqz9g"
Sep 11 18:10:04.030523 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:04.028433    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:10:04.030523 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028459    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-host-kubelet\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.030523 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028485    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-log-socket\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.030523 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:04.028499    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs podName:6609afb5-5000-42c3-814e-9f9ee86a32d4 nodeName:}" failed. No retries permitted until 2026-09-11 18:10:04.528481179 +0000 UTC m=+3.129842336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs") pod "network-metrics-daemon-44nnd" (UID: "6609afb5-5000-42c3-814e-9f9ee86a32d4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:10:04.030523 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028502    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fe7af037-cf3e-4ae8-bdf0-f458a249950f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-759db\" (UID: \"fe7af037-cf3e-4ae8-bdf0-f458a249950f\") " pod="openshift-multus/multus-additional-cni-plugins-759db"
Sep 11 18:10:04.030523 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028532    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-log-socket\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.031444 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028538    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnjj\" (UniqueName: \"kubernetes.io/projected/c882403c-4f90-4d58-a269-9b897f0ba2e8-kube-api-access-nxnjj\") pod \"network-check-target-qhvvw\" (UID: \"c882403c-4f90-4d58-a269-9b897f0ba2e8\") " pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:04.031444 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028575    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c70edc6-7469-4d7c-b3f1-5d88caac76aa-serviceca\") pod \"node-ca-ckcv2\" (UID: \"9c70edc6-7469-4d7c-b3f1-5d88caac76aa\") " pod="openshift-image-registry/node-ca-ckcv2"
Sep 11 18:10:04.031444 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028581    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-run-ovn\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.031444 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028589    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-host-kubelet\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.031444 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028582    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fe7af037-cf3e-4ae8-bdf0-f458a249950f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-759db\" (UID: \"fe7af037-cf3e-4ae8-bdf0-f458a249950f\") " pod="openshift-multus/multus-additional-cni-plugins-759db"
Sep 11 18:10:04.031444 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028601    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-host-slash\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.031444 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028637    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-var-lib-openvswitch\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.031444 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028643    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-host-slash\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.031444 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028719    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-etc-openvswitch\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.031444 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028760    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-host-run-ovn-kubernetes\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.031444 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028809    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-host-run-ovn-kubernetes\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.031444 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028802    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.031444 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028851    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.031444 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028770    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-etc-openvswitch\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.031444 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028853    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe7af037-cf3e-4ae8-bdf0-f458a249950f-system-cni-dir\") pod \"multus-additional-cni-plugins-759db\" (UID: \"fe7af037-cf3e-4ae8-bdf0-f458a249950f\") " pod="openshift-multus/multus-additional-cni-plugins-759db"
Sep 11 18:10:04.031444 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028892    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe7af037-cf3e-4ae8-bdf0-f458a249950f-system-cni-dir\") pod \"multus-additional-cni-plugins-759db\" (UID: \"fe7af037-cf3e-4ae8-bdf0-f458a249950f\") " pod="openshift-multus/multus-additional-cni-plugins-759db"
Sep 11 18:10:04.031444 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028927    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-host-cni-bin\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.032317 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.028985    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb6146c9-7f2c-43a4-8e60-3e995a078646-ovn-node-metrics-cert\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.032317 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.029012    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktm2t\" (UniqueName: \"kubernetes.io/projected/6609afb5-5000-42c3-814e-9f9ee86a32d4-kube-api-access-ktm2t\") pod \"network-metrics-daemon-44nnd\" (UID: \"6609afb5-5000-42c3-814e-9f9ee86a32d4\") " pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:04.032317 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.029044    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe7af037-cf3e-4ae8-bdf0-f458a249950f-cnibin\") pod \"multus-additional-cni-plugins-759db\" (UID: \"fe7af037-cf3e-4ae8-bdf0-f458a249950f\") " pod="openshift-multus/multus-additional-cni-plugins-759db"
Sep 11 18:10:04.032317 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.029050    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c70edc6-7469-4d7c-b3f1-5d88caac76aa-serviceca\") pod \"node-ca-ckcv2\" (UID: \"9c70edc6-7469-4d7c-b3f1-5d88caac76aa\") " pod="openshift-image-registry/node-ca-ckcv2"
Sep 11 18:10:04.032317 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.029454    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cb6146c9-7f2c-43a4-8e60-3e995a078646-env-overrides\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.032317 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.029460    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb6146c9-7f2c-43a4-8e60-3e995a078646-host-cni-bin\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.032317 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.029540    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe7af037-cf3e-4ae8-bdf0-f458a249950f-cnibin\") pod \"multus-additional-cni-plugins-759db\" (UID: \"fe7af037-cf3e-4ae8-bdf0-f458a249950f\") " pod="openshift-multus/multus-additional-cni-plugins-759db"
Sep 11 18:10:04.032679 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.032437    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232-agent-certs\") pod \"konnectivity-agent-j7jrb\" (UID: \"f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232\") " pod="kube-system/konnectivity-agent-j7jrb"
Sep 11 18:10:04.032679 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.032655    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb6146c9-7f2c-43a4-8e60-3e995a078646-ovn-node-metrics-cert\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.042003 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:04.041864    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:10:04.042003 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:04.041904    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:10:04.042003 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:04.041919    2576 projected.go:196] Error preparing data for projected volume kube-api-access-nxnjj for pod openshift-network-diagnostics/network-check-target-qhvvw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:10:04.042003 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:04.041993    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c882403c-4f90-4d58-a269-9b897f0ba2e8-kube-api-access-nxnjj podName:c882403c-4f90-4d58-a269-9b897f0ba2e8 nodeName:}" failed. No retries permitted until 2026-09-11 18:10:04.541972967 +0000 UTC m=+3.143334146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nxnjj" (UniqueName: "kubernetes.io/projected/c882403c-4f90-4d58-a269-9b897f0ba2e8-kube-api-access-nxnjj") pod "network-check-target-qhvvw" (UID: "c882403c-4f90-4d58-a269-9b897f0ba2e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:10:04.047907 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.047830    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb4nf\" (UniqueName: \"kubernetes.io/projected/e0a9600b-c5a4-4e01-99b5-fe0598a4cb52-kube-api-access-fb4nf\") pod \"node-resolver-nqz9g\" (UID: \"e0a9600b-c5a4-4e01-99b5-fe0598a4cb52\") " pod="openshift-dns/node-resolver-nqz9g"
Sep 11 18:10:04.048387 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.048340    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwgbq\" (UniqueName: \"kubernetes.io/projected/cb6146c9-7f2c-43a4-8e60-3e995a078646-kube-api-access-nwgbq\") pod \"ovnkube-node-rxclq\" (UID: \"cb6146c9-7f2c-43a4-8e60-3e995a078646\") " pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.049549 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.049522    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktm2t\" (UniqueName: \"kubernetes.io/projected/6609afb5-5000-42c3-814e-9f9ee86a32d4-kube-api-access-ktm2t\") pod \"network-metrics-daemon-44nnd\" (UID: \"6609afb5-5000-42c3-814e-9f9ee86a32d4\") " pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:04.051015 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.050986    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6wnq\" (UniqueName: \"kubernetes.io/projected/9c70edc6-7469-4d7c-b3f1-5d88caac76aa-kube-api-access-f6wnq\") pod \"node-ca-ckcv2\" (UID: \"9c70edc6-7469-4d7c-b3f1-5d88caac76aa\") " pod="openshift-image-registry/node-ca-ckcv2"
Sep 11 18:10:04.051476 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.051455    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg7bv\" (UniqueName: \"kubernetes.io/projected/fe7af037-cf3e-4ae8-bdf0-f458a249950f-kube-api-access-kg7bv\") pod \"multus-additional-cni-plugins-759db\" (UID: \"fe7af037-cf3e-4ae8-bdf0-f458a249950f\") " pod="openshift-multus/multus-additional-cni-plugins-759db"
Sep 11 18:10:04.115844 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.115805    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jkms9"
Sep 11 18:10:04.125280 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.125235    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7967x"
Sep 11 18:10:04.143823 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.143788    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx"
Sep 11 18:10:04.151559 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.151529    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r4w8t"
Sep 11 18:10:04.160803 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.160777    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nqz9g"
Sep 11 18:10:04.170980 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.170953    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ckcv2"
Sep 11 18:10:04.180218 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.180194    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j7jrb"
Sep 11 18:10:04.180480 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.180458    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:10:04.189275 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.189248    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:04.196222 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.196192    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-759db"
Sep 11 18:10:04.533578 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.533532    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99a9405b-bada-4d50-a3cb-3cf9eb701eee-original-pull-secret\") pod \"global-pull-secret-syncer-db9kz\" (UID: \"99a9405b-bada-4d50-a3cb-3cf9eb701eee\") " pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:04.533786 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.533619    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs\") pod \"network-metrics-daemon-44nnd\" (UID: \"6609afb5-5000-42c3-814e-9f9ee86a32d4\") " pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:04.533786 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:04.533707    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:10:04.533786 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:04.533779    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:10:04.533976 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:04.533795    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99a9405b-bada-4d50-a3cb-3cf9eb701eee-original-pull-secret podName:99a9405b-bada-4d50-a3cb-3cf9eb701eee nodeName:}" failed. No retries permitted until 2026-09-11 18:10:05.533771771 +0000 UTC m=+4.135132921 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99a9405b-bada-4d50-a3cb-3cf9eb701eee-original-pull-secret") pod "global-pull-secret-syncer-db9kz" (UID: "99a9405b-bada-4d50-a3cb-3cf9eb701eee") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:10:04.533976 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:04.533830    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs podName:6609afb5-5000-42c3-814e-9f9ee86a32d4 nodeName:}" failed. No retries permitted until 2026-09-11 18:10:05.533814366 +0000 UTC m=+4.135175514 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs") pod "network-metrics-daemon-44nnd" (UID: "6609afb5-5000-42c3-814e-9f9ee86a32d4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:10:04.634951 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.634902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnjj\" (UniqueName: \"kubernetes.io/projected/c882403c-4f90-4d58-a269-9b897f0ba2e8-kube-api-access-nxnjj\") pod \"network-check-target-qhvvw\" (UID: \"c882403c-4f90-4d58-a269-9b897f0ba2e8\") " pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:04.635154 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:04.635094    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:10:04.635154 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:04.635127    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:10:04.635154 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:04.635142    2576 projected.go:196] Error preparing data for projected volume kube-api-access-nxnjj for pod openshift-network-diagnostics/network-check-target-qhvvw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:10:04.635304 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:04.635204    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c882403c-4f90-4d58-a269-9b897f0ba2e8-kube-api-access-nxnjj podName:c882403c-4f90-4d58-a269-9b897f0ba2e8 nodeName:}" failed. No retries permitted until 2026-09-11 18:10:05.635186655 +0000 UTC m=+4.236547797 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxnjj" (UniqueName: "kubernetes.io/projected/c882403c-4f90-4d58-a269-9b897f0ba2e8-kube-api-access-nxnjj") pod "network-check-target-qhvvw" (UID: "c882403c-4f90-4d58-a269-9b897f0ba2e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:10:04.869074 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.868980    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 18:05:04 +0000 UTC" deadline="2028-03-16 14:09:52.93899627 +0000 UTC"
Sep 11 18:10:04.869074 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:04.869020    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13243h59m48.069980182s"
Sep 11 18:10:05.162822 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:05.162793    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a9600b_c5a4_4e01_99b5_fe0598a4cb52.slice/crio-a8133c0f44d7e8125755ee61321eacfdc84437f56bc55da0cb958094837de229 WatchSource:0}: Error finding container a8133c0f44d7e8125755ee61321eacfdc84437f56bc55da0cb958094837de229: Status 404 returned error can't find the container with id a8133c0f44d7e8125755ee61321eacfdc84437f56bc55da0cb958094837de229
Sep 11 18:10:05.164175 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:05.163700    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6146c9_7f2c_43a4_8e60_3e995a078646.slice/crio-6531a7e676a6291a46d9a4f02513a2d18285adbcb84539f5f4837ec9ed6edecc WatchSource:0}: Error finding container 6531a7e676a6291a46d9a4f02513a2d18285adbcb84539f5f4837ec9ed6edecc: Status 404 returned error can't find the container with id 6531a7e676a6291a46d9a4f02513a2d18285adbcb84539f5f4837ec9ed6edecc
Sep 11 18:10:05.164513 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:05.164486    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a91968_3ccc_4258_b71e_f98563bf0138.slice/crio-61b3a5d3bff6911ec6aa17f918db159869bf059a3eae0cfe0e7003e973c92b20 WatchSource:0}: Error finding container 61b3a5d3bff6911ec6aa17f918db159869bf059a3eae0cfe0e7003e973c92b20: Status 404 returned error can't find the container with id 61b3a5d3bff6911ec6aa17f918db159869bf059a3eae0cfe0e7003e973c92b20
Sep 11 18:10:05.544362 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:05.544158    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99a9405b-bada-4d50-a3cb-3cf9eb701eee-original-pull-secret\") pod \"global-pull-secret-syncer-db9kz\" (UID: \"99a9405b-bada-4d50-a3cb-3cf9eb701eee\") " pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:05.544362 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:05.544343    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:10:05.544362 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:05.544369    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs\") pod \"network-metrics-daemon-44nnd\" (UID: \"6609afb5-5000-42c3-814e-9f9ee86a32d4\") " pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:05.544590 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:05.544420    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99a9405b-bada-4d50-a3cb-3cf9eb701eee-original-pull-secret podName:99a9405b-bada-4d50-a3cb-3cf9eb701eee nodeName:}" failed. No retries permitted until 2026-09-11 18:10:07.544405418 +0000 UTC m=+6.145766576 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99a9405b-bada-4d50-a3cb-3cf9eb701eee-original-pull-secret") pod "global-pull-secret-syncer-db9kz" (UID: "99a9405b-bada-4d50-a3cb-3cf9eb701eee") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:10:05.544590 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:05.544473    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:10:05.544590 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:05.544519    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs podName:6609afb5-5000-42c3-814e-9f9ee86a32d4 nodeName:}" failed. No retries permitted until 2026-09-11 18:10:07.544507417 +0000 UTC m=+6.145868565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs") pod "network-metrics-daemon-44nnd" (UID: "6609afb5-5000-42c3-814e-9f9ee86a32d4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:10:05.645218 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:05.645173    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnjj\" (UniqueName: \"kubernetes.io/projected/c882403c-4f90-4d58-a269-9b897f0ba2e8-kube-api-access-nxnjj\") pod \"network-check-target-qhvvw\" (UID: \"c882403c-4f90-4d58-a269-9b897f0ba2e8\") " pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:05.645397 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:05.645381    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:10:05.645471 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:05.645407    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:10:05.645471 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:05.645418    2576 projected.go:196] Error preparing data for projected volume kube-api-access-nxnjj for pod openshift-network-diagnostics/network-check-target-qhvvw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:10:05.645581 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:05.645483    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c882403c-4f90-4d58-a269-9b897f0ba2e8-kube-api-access-nxnjj podName:c882403c-4f90-4d58-a269-9b897f0ba2e8 nodeName:}" failed. No retries permitted until 2026-09-11 18:10:07.645463131 +0000 UTC m=+6.246824277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxnjj" (UniqueName: "kubernetes.io/projected/c882403c-4f90-4d58-a269-9b897f0ba2e8-kube-api-access-nxnjj") pod "network-check-target-qhvvw" (UID: "c882403c-4f90-4d58-a269-9b897f0ba2e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:10:05.870077 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:05.870030    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 18:05:04 +0000 UTC" deadline="2028-02-25 00:27:07.82968984 +0000 UTC"
Sep 11 18:10:05.870077 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:05.870073    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12750h17m1.959621405s"
Sep 11 18:10:05.920878 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:05.920835    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:05.921054 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:05.920851    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:05.922727 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:05.921944    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44nnd" podUID="6609afb5-5000-42c3-814e-9f9ee86a32d4"
Sep 11 18:10:05.922727 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:05.922231    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db9kz" podUID="99a9405b-bada-4d50-a3cb-3cf9eb701eee"
Sep 11 18:10:05.922727 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:05.922299    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:05.922727 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:05.922404    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvvw" podUID="c882403c-4f90-4d58-a269-9b897f0ba2e8"
Sep 11 18:10:05.927432 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:05.927362    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r4w8t" event={"ID":"63b887df-53bd-4441-a220-b5d82b09bbbf","Type":"ContainerStarted","Data":"bf5de6be34520ba659537f2576299b81305d37e22145b4a6dd4fe749a6636408"}
Sep 11 18:10:05.929684 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:05.929604    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7967x" event={"ID":"4fef7440-cfca-4eaf-a0f9-c102ea506ab8","Type":"ContainerStarted","Data":"be693068f65f898ece3b45b741bdd09e113a7b61052be3f7e5b1dcd3e947e54e"}
Sep 11 18:10:05.934878 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:05.934851    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ckcv2" event={"ID":"9c70edc6-7469-4d7c-b3f1-5d88caac76aa","Type":"ContainerStarted","Data":"0df027f846075102eb3404a1502eb212d4ce6723615a945ca39c4621669a1b7d"}
Sep 11 18:10:05.945660 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:05.945626    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jkms9" event={"ID":"01a91968-3ccc-4258-b71e-f98563bf0138","Type":"ContainerStarted","Data":"61b3a5d3bff6911ec6aa17f918db159869bf059a3eae0cfe0e7003e973c92b20"}
Sep 11 18:10:05.948828 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:05.948796    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxclq" event={"ID":"cb6146c9-7f2c-43a4-8e60-3e995a078646","Type":"ContainerStarted","Data":"6531a7e676a6291a46d9a4f02513a2d18285adbcb84539f5f4837ec9ed6edecc"}
Sep 11 18:10:05.966115 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:05.966080    2576 generic.go:356] "Generic (PLEG): container finished" podID="8471b73601d05635d4bf15db7b2236da" containerID="4f6c4d92804f259a882594b1dc208b99be739eb64199da1ae8d183da97e12480" exitCode=0
Sep 11 18:10:05.966434 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:05.966366    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-111.ec2.internal" event={"ID":"8471b73601d05635d4bf15db7b2236da","Type":"ContainerDied","Data":"4f6c4d92804f259a882594b1dc208b99be739eb64199da1ae8d183da97e12480"}
Sep 11 18:10:05.973833 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:05.973804    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j7jrb" event={"ID":"f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232","Type":"ContainerStarted","Data":"80574d5a314bddf00eaf99fb9b3ecf017eb13262d6e1781b80453555d1c1bf7b"}
Sep 11 18:10:05.976199 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:05.976171    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx" event={"ID":"dc85a151-5b62-439d-98e1-8a117c5a2f3b","Type":"ContainerStarted","Data":"d7a5f2d19c38eb417a3eaeb21fd6351ba32a80a3a507cbcc122700dc985dc9d1"}
Sep 11 18:10:05.981704 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:05.981679    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-759db" event={"ID":"fe7af037-cf3e-4ae8-bdf0-f458a249950f","Type":"ContainerStarted","Data":"0234656555bae3624f112fd6d331fc4c0d75b2b9e776947858645857c238b351"}
Sep 11 18:10:05.994893 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:05.989647    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nqz9g" event={"ID":"e0a9600b-c5a4-4e01-99b5-fe0598a4cb52","Type":"ContainerStarted","Data":"a8133c0f44d7e8125755ee61321eacfdc84437f56bc55da0cb958094837de229"}
Sep 11 18:10:05.998930 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:05.996308    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-111.ec2.internal" event={"ID":"d6fbb6fad8f6855b4beb91cec165f915","Type":"ContainerStarted","Data":"d373579bcff530ccab121ec5b063e51b1891f9946c37c4206ad5ce6719e2ce69"}
Sep 11 18:10:07.019962 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:07.019841    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-111.ec2.internal" event={"ID":"8471b73601d05635d4bf15db7b2236da","Type":"ContainerStarted","Data":"a942e2fe82d1d97eb0269457b10fd7f97575f1d12e26802e35c0ca8d59d417c7"}
Sep 11 18:10:07.060387 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:07.059505    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-111.ec2.internal" podStartSLOduration=4.059478929 podStartE2EDuration="4.059478929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:10:06.023823703 +0000 UTC m=+4.625184873" watchObservedRunningTime="2026-09-11 18:10:07.059478929 +0000 UTC m=+5.660840097"
Sep 11 18:10:07.562749 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:07.562708    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99a9405b-bada-4d50-a3cb-3cf9eb701eee-original-pull-secret\") pod \"global-pull-secret-syncer-db9kz\" (UID: \"99a9405b-bada-4d50-a3cb-3cf9eb701eee\") " pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:07.562933 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:07.562767    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs\") pod \"network-metrics-daemon-44nnd\" (UID: \"6609afb5-5000-42c3-814e-9f9ee86a32d4\") " pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:07.563019 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:07.562949    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:10:07.563019 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:07.563013    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs podName:6609afb5-5000-42c3-814e-9f9ee86a32d4 nodeName:}" failed. No retries permitted until 2026-09-11 18:10:11.562993804 +0000 UTC m=+10.164354952 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs") pod "network-metrics-daemon-44nnd" (UID: "6609afb5-5000-42c3-814e-9f9ee86a32d4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:10:07.563460 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:07.563439    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:10:07.563564 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:07.563498    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99a9405b-bada-4d50-a3cb-3cf9eb701eee-original-pull-secret podName:99a9405b-bada-4d50-a3cb-3cf9eb701eee nodeName:}" failed. No retries permitted until 2026-09-11 18:10:11.563481434 +0000 UTC m=+10.164842581 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99a9405b-bada-4d50-a3cb-3cf9eb701eee-original-pull-secret") pod "global-pull-secret-syncer-db9kz" (UID: "99a9405b-bada-4d50-a3cb-3cf9eb701eee") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:10:07.663823 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:07.663788    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnjj\" (UniqueName: \"kubernetes.io/projected/c882403c-4f90-4d58-a269-9b897f0ba2e8-kube-api-access-nxnjj\") pod \"network-check-target-qhvvw\" (UID: \"c882403c-4f90-4d58-a269-9b897f0ba2e8\") " pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:07.664005 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:07.663980    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:10:07.664005 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:07.664003    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:10:07.664121 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:07.664016    2576 projected.go:196] Error preparing data for projected volume kube-api-access-nxnjj for pod openshift-network-diagnostics/network-check-target-qhvvw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:10:07.664121 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:07.664077    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c882403c-4f90-4d58-a269-9b897f0ba2e8-kube-api-access-nxnjj podName:c882403c-4f90-4d58-a269-9b897f0ba2e8 nodeName:}" failed. No retries permitted until 2026-09-11 18:10:11.664057396 +0000 UTC m=+10.265418557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxnjj" (UniqueName: "kubernetes.io/projected/c882403c-4f90-4d58-a269-9b897f0ba2e8-kube-api-access-nxnjj") pod "network-check-target-qhvvw" (UID: "c882403c-4f90-4d58-a269-9b897f0ba2e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:10:07.905676 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:07.904767    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:07.905676 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:07.904916    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db9kz" podUID="99a9405b-bada-4d50-a3cb-3cf9eb701eee"
Sep 11 18:10:07.905676 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:07.905325    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:07.905676 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:07.905438    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44nnd" podUID="6609afb5-5000-42c3-814e-9f9ee86a32d4"
Sep 11 18:10:07.905676 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:07.905498    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:07.905676 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:07.905582    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvvw" podUID="c882403c-4f90-4d58-a269-9b897f0ba2e8"
Sep 11 18:10:09.904953 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:09.904432    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:09.904953 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:09.904563    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:09.904953 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:09.904567    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvvw" podUID="c882403c-4f90-4d58-a269-9b897f0ba2e8"
Sep 11 18:10:09.904953 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:09.904443    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:09.904953 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:09.904707    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db9kz" podUID="99a9405b-bada-4d50-a3cb-3cf9eb701eee"
Sep 11 18:10:09.904953 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:09.904850    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44nnd" podUID="6609afb5-5000-42c3-814e-9f9ee86a32d4"
Sep 11 18:10:11.596786 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:11.596743    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99a9405b-bada-4d50-a3cb-3cf9eb701eee-original-pull-secret\") pod \"global-pull-secret-syncer-db9kz\" (UID: \"99a9405b-bada-4d50-a3cb-3cf9eb701eee\") " pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:11.597250 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:11.596797    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs\") pod \"network-metrics-daemon-44nnd\" (UID: \"6609afb5-5000-42c3-814e-9f9ee86a32d4\") " pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:11.597250 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:11.596877    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:10:11.597250 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:11.596913    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:10:11.597250 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:11.596948    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99a9405b-bada-4d50-a3cb-3cf9eb701eee-original-pull-secret podName:99a9405b-bada-4d50-a3cb-3cf9eb701eee nodeName:}" failed. No retries permitted until 2026-09-11 18:10:19.596928541 +0000 UTC m=+18.198289695 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99a9405b-bada-4d50-a3cb-3cf9eb701eee-original-pull-secret") pod "global-pull-secret-syncer-db9kz" (UID: "99a9405b-bada-4d50-a3cb-3cf9eb701eee") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:10:11.597250 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:11.596979    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs podName:6609afb5-5000-42c3-814e-9f9ee86a32d4 nodeName:}" failed. No retries permitted until 2026-09-11 18:10:19.596959688 +0000 UTC m=+18.198320842 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs") pod "network-metrics-daemon-44nnd" (UID: "6609afb5-5000-42c3-814e-9f9ee86a32d4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:10:11.697444 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:11.697412    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnjj\" (UniqueName: \"kubernetes.io/projected/c882403c-4f90-4d58-a269-9b897f0ba2e8-kube-api-access-nxnjj\") pod \"network-check-target-qhvvw\" (UID: \"c882403c-4f90-4d58-a269-9b897f0ba2e8\") " pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:11.697723 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:11.697606    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:10:11.697723 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:11.697638    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:10:11.697723 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:11.697652    2576 projected.go:196] Error preparing data for projected volume kube-api-access-nxnjj for pod openshift-network-diagnostics/network-check-target-qhvvw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:10:11.697723 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:11.697714    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c882403c-4f90-4d58-a269-9b897f0ba2e8-kube-api-access-nxnjj podName:c882403c-4f90-4d58-a269-9b897f0ba2e8 nodeName:}" failed. No retries permitted until 2026-09-11 18:10:19.697697065 +0000 UTC m=+18.299058219 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxnjj" (UniqueName: "kubernetes.io/projected/c882403c-4f90-4d58-a269-9b897f0ba2e8-kube-api-access-nxnjj") pod "network-check-target-qhvvw" (UID: "c882403c-4f90-4d58-a269-9b897f0ba2e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:10:11.905604 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:11.905505    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:11.905604 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:11.905600    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:11.905815 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:11.905645    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:11.905815 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:11.905676    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvvw" podUID="c882403c-4f90-4d58-a269-9b897f0ba2e8"
Sep 11 18:10:11.905815 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:11.905732    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44nnd" podUID="6609afb5-5000-42c3-814e-9f9ee86a32d4"
Sep 11 18:10:11.905815 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:11.905809    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db9kz" podUID="99a9405b-bada-4d50-a3cb-3cf9eb701eee"
Sep 11 18:10:13.905560 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:13.904949    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:13.905560 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:13.905086    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44nnd" podUID="6609afb5-5000-42c3-814e-9f9ee86a32d4"
Sep 11 18:10:13.905560 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:13.905450    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:13.905560 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:13.905523    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvvw" podUID="c882403c-4f90-4d58-a269-9b897f0ba2e8"
Sep 11 18:10:13.906220 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:13.906191    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:13.906324 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:13.906274    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db9kz" podUID="99a9405b-bada-4d50-a3cb-3cf9eb701eee"
Sep 11 18:10:15.904561 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:15.904527    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:15.905018 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:15.904535    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:15.905018 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:15.904668    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db9kz" podUID="99a9405b-bada-4d50-a3cb-3cf9eb701eee"
Sep 11 18:10:15.905018 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:15.904712    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:15.905018 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:15.904812    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44nnd" podUID="6609afb5-5000-42c3-814e-9f9ee86a32d4"
Sep 11 18:10:15.905018 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:15.904898    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvvw" podUID="c882403c-4f90-4d58-a269-9b897f0ba2e8"
Sep 11 18:10:17.904536 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:17.904413    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:17.904536 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:17.904457    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:17.904536 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:17.904539    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:17.905072 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:17.904558    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvvw" podUID="c882403c-4f90-4d58-a269-9b897f0ba2e8"
Sep 11 18:10:17.905072 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:17.904665    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44nnd" podUID="6609afb5-5000-42c3-814e-9f9ee86a32d4"
Sep 11 18:10:17.905072 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:17.904756    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db9kz" podUID="99a9405b-bada-4d50-a3cb-3cf9eb701eee"
Sep 11 18:10:19.658620 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:19.658388    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99a9405b-bada-4d50-a3cb-3cf9eb701eee-original-pull-secret\") pod \"global-pull-secret-syncer-db9kz\" (UID: \"99a9405b-bada-4d50-a3cb-3cf9eb701eee\") " pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:19.659090 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:19.658638    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs\") pod \"network-metrics-daemon-44nnd\" (UID: \"6609afb5-5000-42c3-814e-9f9ee86a32d4\") " pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:19.659090 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:19.658533    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:10:19.659090 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:19.658744    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:10:19.659090 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:19.658750    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99a9405b-bada-4d50-a3cb-3cf9eb701eee-original-pull-secret podName:99a9405b-bada-4d50-a3cb-3cf9eb701eee nodeName:}" failed. No retries permitted until 2026-09-11 18:10:35.658731066 +0000 UTC m=+34.260092226 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99a9405b-bada-4d50-a3cb-3cf9eb701eee-original-pull-secret") pod "global-pull-secret-syncer-db9kz" (UID: "99a9405b-bada-4d50-a3cb-3cf9eb701eee") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:10:19.659090 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:19.658778    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs podName:6609afb5-5000-42c3-814e-9f9ee86a32d4 nodeName:}" failed. No retries permitted until 2026-09-11 18:10:35.658767881 +0000 UTC m=+34.260129032 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs") pod "network-metrics-daemon-44nnd" (UID: "6609afb5-5000-42c3-814e-9f9ee86a32d4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:10:19.759832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:19.759794    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnjj\" (UniqueName: \"kubernetes.io/projected/c882403c-4f90-4d58-a269-9b897f0ba2e8-kube-api-access-nxnjj\") pod \"network-check-target-qhvvw\" (UID: \"c882403c-4f90-4d58-a269-9b897f0ba2e8\") " pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:19.760018 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:19.759974    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:10:19.760018 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:19.759997    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:10:19.760018 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:19.760009    2576 projected.go:196] Error preparing data for projected volume kube-api-access-nxnjj for pod openshift-network-diagnostics/network-check-target-qhvvw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:10:19.760188 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:19.760083    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c882403c-4f90-4d58-a269-9b897f0ba2e8-kube-api-access-nxnjj podName:c882403c-4f90-4d58-a269-9b897f0ba2e8 nodeName:}" failed. No retries permitted until 2026-09-11 18:10:35.760057636 +0000 UTC m=+34.361418792 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxnjj" (UniqueName: "kubernetes.io/projected/c882403c-4f90-4d58-a269-9b897f0ba2e8-kube-api-access-nxnjj") pod "network-check-target-qhvvw" (UID: "c882403c-4f90-4d58-a269-9b897f0ba2e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:10:19.905180 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:19.905142    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:19.905357 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:19.905148    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:19.905357 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:19.905279    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db9kz" podUID="99a9405b-bada-4d50-a3cb-3cf9eb701eee"
Sep 11 18:10:19.905357 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:19.905153    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:19.905503 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:19.905364    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44nnd" podUID="6609afb5-5000-42c3-814e-9f9ee86a32d4"
Sep 11 18:10:19.905503 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:19.905440    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvvw" podUID="c882403c-4f90-4d58-a269-9b897f0ba2e8"
Sep 11 18:10:21.905112 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:21.905074    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:21.905617 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:21.905198    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44nnd" podUID="6609afb5-5000-42c3-814e-9f9ee86a32d4"
Sep 11 18:10:21.905617 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:21.905258    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:21.905617 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:21.905338    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvvw" podUID="c882403c-4f90-4d58-a269-9b897f0ba2e8"
Sep 11 18:10:21.905617 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:21.905347    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:21.905617 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:21.905430    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db9kz" podUID="99a9405b-bada-4d50-a3cb-3cf9eb701eee"
Sep 11 18:10:23.904620 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:23.904351    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:23.905250 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:23.904358    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:23.905250 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:23.904645    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvvw" podUID="c882403c-4f90-4d58-a269-9b897f0ba2e8"
Sep 11 18:10:23.905250 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:23.904745    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44nnd" podUID="6609afb5-5000-42c3-814e-9f9ee86a32d4"
Sep 11 18:10:23.905250 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:23.904358    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:23.905250 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:23.904837    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db9kz" podUID="99a9405b-bada-4d50-a3cb-3cf9eb701eee"
Sep 11 18:10:24.074657 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:24.074620    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r4w8t" event={"ID":"63b887df-53bd-4441-a220-b5d82b09bbbf","Type":"ContainerStarted","Data":"3a27fc57040bba234510949d9baf173a162c5c2ce273e1ca42bf5e967750f087"}
Sep 11 18:10:24.076301 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:24.076274    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ckcv2" event={"ID":"9c70edc6-7469-4d7c-b3f1-5d88caac76aa","Type":"ContainerStarted","Data":"13e19c3c2e1a068ae464aab46d3416f73f45ef22c7fec3923816bd13ba4fb19f"}
Sep 11 18:10:24.077742 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:24.077720    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jkms9" event={"ID":"01a91968-3ccc-4258-b71e-f98563bf0138","Type":"ContainerStarted","Data":"e11d401982b2de024c44adab5e6ea2ea6ffea6ace87e22052b7f00c067102412"}
Sep 11 18:10:24.083351 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:24.083314    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxclq" event={"ID":"cb6146c9-7f2c-43a4-8e60-3e995a078646","Type":"ContainerStarted","Data":"63bb444424c5a7b53a39e2baaef0f774706d59d83f6347c337415e4cd927f911"}
Sep 11 18:10:24.083458 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:24.083407    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxclq" event={"ID":"cb6146c9-7f2c-43a4-8e60-3e995a078646","Type":"ContainerStarted","Data":"31528bb324acc73bab09f55d81369d9d84a462dda72c8aa97a24264c4d03c244"}
Sep 11 18:10:24.083458 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:24.083447    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxclq" event={"ID":"cb6146c9-7f2c-43a4-8e60-3e995a078646","Type":"ContainerStarted","Data":"48c7ae7472eb2279ee02e2b29995517402869de7fa5d58f2a15df54958516dac"}
Sep 11 18:10:24.083554 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:24.083479    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxclq" event={"ID":"cb6146c9-7f2c-43a4-8e60-3e995a078646","Type":"ContainerStarted","Data":"7a3d3065f5f99c582f69b35c0a2a77854d349aab95cb7dc23765f4510c4a5e25"}
Sep 11 18:10:24.083554 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:24.083513    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxclq" event={"ID":"cb6146c9-7f2c-43a4-8e60-3e995a078646","Type":"ContainerStarted","Data":"4224615434d2039eb3720e550a50fe1f475538d7880cc3ecfe759f1f6406428c"}
Sep 11 18:10:24.083658 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:24.083556    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxclq" event={"ID":"cb6146c9-7f2c-43a4-8e60-3e995a078646","Type":"ContainerStarted","Data":"578c37ec1a86e327daea58c67a5653324b5652f3f7e709486d79d500499ff706"}
Sep 11 18:10:24.084678 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:24.084648    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j7jrb" event={"ID":"f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232","Type":"ContainerStarted","Data":"d9abf1fb77adebffd0617c973e640184c2fa19e36fabad901654ebb24ec46405"}
Sep 11 18:10:24.085963 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:24.085946    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx" event={"ID":"dc85a151-5b62-439d-98e1-8a117c5a2f3b","Type":"ContainerStarted","Data":"c3de722c6c59be4144f64090187089c6398b5716431f23329ccae5feb17c63f1"}
Sep 11 18:10:24.087230 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:24.087210    2576 generic.go:356] "Generic (PLEG): container finished" podID="fe7af037-cf3e-4ae8-bdf0-f458a249950f" containerID="c36fe565f977e0a7e2aaeb2ee7ea7d6a11e97a15e7346ce895ed3f91d3af9611" exitCode=0
Sep 11 18:10:24.087309 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:24.087286    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-759db" event={"ID":"fe7af037-cf3e-4ae8-bdf0-f458a249950f","Type":"ContainerDied","Data":"c36fe565f977e0a7e2aaeb2ee7ea7d6a11e97a15e7346ce895ed3f91d3af9611"}
Sep 11 18:10:24.088732 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:24.088708    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nqz9g" event={"ID":"e0a9600b-c5a4-4e01-99b5-fe0598a4cb52","Type":"ContainerStarted","Data":"112a156185b974e6d514feacbc3c01e3d98b686e176e6f183093cae9b95ee979"}
Sep 11 18:10:24.102983 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:24.102941    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-111.ec2.internal" podStartSLOduration=21.102925838 podStartE2EDuration="21.102925838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:10:07.060368243 +0000 UTC m=+5.661729412" watchObservedRunningTime="2026-09-11 18:10:24.102925838 +0000 UTC m=+22.704287002"
Sep 11 18:10:24.126852 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:24.126805    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-r4w8t" podStartSLOduration=4.375897836 podStartE2EDuration="22.126790449s" totalImagesPullingTime="17.750892613s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:10:24.102726902 +0000 UTC m=+22.704088061" watchObservedRunningTime="2026-09-11 18:10:24.126790449 +0000 UTC m=+22.728151612"
Sep 11 18:10:24.127005 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:24.126899    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nqz9g" podStartSLOduration=4.358469621 podStartE2EDuration="22.126894524s" totalImagesPullingTime="17.768424903s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:10:24.123267038 +0000 UTC m=+22.724628204" watchObservedRunningTime="2026-09-11 18:10:24.126894524 +0000 UTC m=+22.728255687"
Sep 11 18:10:24.179060 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:24.178945    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ckcv2" podStartSLOduration=11.56482868 podStartE2EDuration="22.178924323s" totalImagesPullingTime="10.614095643s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:10:24.150787822 +0000 UTC m=+22.752148987" watchObservedRunningTime="2026-09-11 18:10:24.178924323 +0000 UTC m=+22.780285487"
Sep 11 18:10:24.233595 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:24.233533    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-j7jrb" podStartSLOduration=11.628976014 podStartE2EDuration="22.233516856s" totalImagesPullingTime="10.604540842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:10:24.188833045 +0000 UTC m=+22.790194215" watchObservedRunningTime="2026-09-11 18:10:24.233516856 +0000 UTC m=+22.834878019"
Sep 11 18:10:24.259600 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:24.259535    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jkms9" podStartSLOduration=4.440376248 podStartE2EDuration="22.259513153s" totalImagesPullingTime="17.819136905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:10:24.258352278 +0000 UTC m=+22.859713443" watchObservedRunningTime="2026-09-11 18:10:24.259513153 +0000 UTC m=+22.860874317"
Sep 11 18:10:24.761311 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:24.761125    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 18:10:24.890826 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:24.890637    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T18:10:24.761295153Z","UUID":"32bf2c0e-9407-49a7-8cbc-77eda724aa0b","Handler":null,"Name":"","Endpoint":""}
Sep 11 18:10:24.892032 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:24.892013    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 18:10:24.892157 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:24.892047    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 18:10:25.092940 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:25.092903    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7967x" event={"ID":"4fef7440-cfca-4eaf-a0f9-c102ea506ab8","Type":"ContainerStarted","Data":"dc3b22dac53cc6212be4ef9269e2925cbc43cce109c4bc425460454d698c9cbc"}
Sep 11 18:10:25.097085 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:25.097042    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx" event={"ID":"dc85a151-5b62-439d-98e1-8a117c5a2f3b","Type":"ContainerStarted","Data":"2911f981b59c45ad8250cb815d4cc4b11a3e68d30b053c17d76e0c20a4edb748"}
Sep 11 18:10:25.771289 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:25.771152    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-j7jrb"
Sep 11 18:10:25.772018 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:25.771982    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-j7jrb"
Sep 11 18:10:25.792955 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:25.792905    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7967x" podStartSLOduration=6.042148416 podStartE2EDuration="23.792890664s" totalImagesPullingTime="17.750742248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:10:25.111502582 +0000 UTC m=+23.712863747" watchObservedRunningTime="2026-09-11 18:10:25.792890664 +0000 UTC m=+24.394251824"
Sep 11 18:10:25.905279 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:25.905195    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:25.905435 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:25.905202    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:25.905435 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:25.905328    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:25.905435 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:25.905327    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db9kz" podUID="99a9405b-bada-4d50-a3cb-3cf9eb701eee"
Sep 11 18:10:25.905435 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:25.905422    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44nnd" podUID="6609afb5-5000-42c3-814e-9f9ee86a32d4"
Sep 11 18:10:25.905678 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:25.905497    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvvw" podUID="c882403c-4f90-4d58-a269-9b897f0ba2e8"
Sep 11 18:10:26.104578 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:26.104530    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxclq" event={"ID":"cb6146c9-7f2c-43a4-8e60-3e995a078646","Type":"ContainerStarted","Data":"f00305a32b7b04e3730acd1dd67706008db42b6e1c1443ae3a0ad11e5bd54a5c"}
Sep 11 18:10:27.109010 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:27.108972    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx" event={"ID":"dc85a151-5b62-439d-98e1-8a117c5a2f3b","Type":"ContainerStarted","Data":"470dc4fc31075772ecfc7f402dfc3f0d192d8d19954476c23ce304f2e069aac7"}
Sep 11 18:10:27.109507 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:27.108991    2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 18:10:27.904908 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:27.904873    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:27.905107 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:27.904886    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:27.905107 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:27.904996    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvvw" podUID="c882403c-4f90-4d58-a269-9b897f0ba2e8"
Sep 11 18:10:27.905107 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:27.904892    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:27.905107 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:27.905073    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db9kz" podUID="99a9405b-bada-4d50-a3cb-3cf9eb701eee"
Sep 11 18:10:27.905308 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:27.905150    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44nnd" podUID="6609afb5-5000-42c3-814e-9f9ee86a32d4"
Sep 11 18:10:29.115435 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:29.115248    2576 generic.go:356] "Generic (PLEG): container finished" podID="fe7af037-cf3e-4ae8-bdf0-f458a249950f" containerID="980e5523e97e428661d16d8c04dd47f65283ca4fac73d167a24c07fe2da08a11" exitCode=0
Sep 11 18:10:29.116039 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:29.115337    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-759db" event={"ID":"fe7af037-cf3e-4ae8-bdf0-f458a249950f","Type":"ContainerDied","Data":"980e5523e97e428661d16d8c04dd47f65283ca4fac73d167a24c07fe2da08a11"}
Sep 11 18:10:29.119288 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:29.119260    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rxclq" event={"ID":"cb6146c9-7f2c-43a4-8e60-3e995a078646","Type":"ContainerStarted","Data":"dea6fd3dceb691a2b1775ede5518ef64fac0ebb3cd02e97a13e14f0b68360136"}
Sep 11 18:10:29.119492 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:29.119474    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:29.119627 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:29.119563    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:29.134858 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:29.134832    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:29.134987 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:29.134933    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:29.143513 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:29.143471    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx8qx" podStartSLOduration=6.289097261 podStartE2EDuration="27.143457162s" totalImagesPullingTime="20.854359901s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:10:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:10:27.139804442 +0000 UTC m=+25.741165625" watchObservedRunningTime="2026-09-11 18:10:29.143457162 +0000 UTC m=+27.744818324"
Sep 11 18:10:29.182159 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:29.182111    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rxclq" podStartSLOduration=9.356831504 podStartE2EDuration="27.182094669s" totalImagesPullingTime="17.825263165s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:10:29.181848083 +0000 UTC m=+27.783209237" watchObservedRunningTime="2026-09-11 18:10:29.182094669 +0000 UTC m=+27.783455833"
Sep 11 18:10:29.908444 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:29.908415    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:29.908444 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:29.908428    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:29.908444 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:29.908419    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:29.908751 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:29.908527    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db9kz" podUID="99a9405b-bada-4d50-a3cb-3cf9eb701eee"
Sep 11 18:10:29.908751 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:29.908617    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvvw" podUID="c882403c-4f90-4d58-a269-9b897f0ba2e8"
Sep 11 18:10:29.908751 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:29.908730    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44nnd" podUID="6609afb5-5000-42c3-814e-9f9ee86a32d4"
Sep 11 18:10:30.123488 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:30.123452    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-759db" event={"ID":"fe7af037-cf3e-4ae8-bdf0-f458a249950f","Type":"ContainerStarted","Data":"f972d10cea3ae43edc0c1c916640e5920c12200f8cf468b4f7c6ad5edb6a6cab"}
Sep 11 18:10:30.124224 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:30.123524    2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 18:10:30.646541 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:30.646465    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qhvvw"]
Sep 11 18:10:30.646700 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:30.646615    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:30.646750 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:30.646697    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvvw" podUID="c882403c-4f90-4d58-a269-9b897f0ba2e8"
Sep 11 18:10:30.650019 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:30.649993    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-db9kz"]
Sep 11 18:10:30.650138 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:30.650078    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:30.650185 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:30.650148    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db9kz" podUID="99a9405b-bada-4d50-a3cb-3cf9eb701eee"
Sep 11 18:10:30.656496 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:30.656471    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-44nnd"]
Sep 11 18:10:30.656576 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:30.656561    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:30.656671 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:30.656654    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44nnd" podUID="6609afb5-5000-42c3-814e-9f9ee86a32d4"
Sep 11 18:10:31.128662 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:31.128451    2576 generic.go:356] "Generic (PLEG): container finished" podID="fe7af037-cf3e-4ae8-bdf0-f458a249950f" containerID="f972d10cea3ae43edc0c1c916640e5920c12200f8cf468b4f7c6ad5edb6a6cab" exitCode=0
Sep 11 18:10:31.129238 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:31.128753    2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 18:10:31.129238 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:31.128826    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-759db" event={"ID":"fe7af037-cf3e-4ae8-bdf0-f458a249950f","Type":"ContainerDied","Data":"f972d10cea3ae43edc0c1c916640e5920c12200f8cf468b4f7c6ad5edb6a6cab"}
Sep 11 18:10:31.908349 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:31.908158    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:31.908506 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:31.908433    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvvw" podUID="c882403c-4f90-4d58-a269-9b897f0ba2e8"
Sep 11 18:10:32.134004 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:32.133857    2576 generic.go:356] "Generic (PLEG): container finished" podID="fe7af037-cf3e-4ae8-bdf0-f458a249950f" containerID="5ba5afc8c77cde318c81da24a5b2ed014b7b74291020b37f615b9c850630485c" exitCode=0
Sep 11 18:10:32.134004 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:32.133903    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-759db" event={"ID":"fe7af037-cf3e-4ae8-bdf0-f458a249950f","Type":"ContainerDied","Data":"5ba5afc8c77cde318c81da24a5b2ed014b7b74291020b37f615b9c850630485c"}
Sep 11 18:10:32.904524 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:32.904489    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:32.904727 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:32.904642    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44nnd" podUID="6609afb5-5000-42c3-814e-9f9ee86a32d4"
Sep 11 18:10:32.904727 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:32.904708    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:32.904839 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:32.904813    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db9kz" podUID="99a9405b-bada-4d50-a3cb-3cf9eb701eee"
Sep 11 18:10:33.074298 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:33.074222    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:10:33.074466 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:33.074418    2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 18:10:33.095910 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:33.095873    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-rxclq" podUID="cb6146c9-7f2c-43a4-8e60-3e995a078646" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 11 18:10:33.104648 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:33.104614    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-rxclq" podUID="cb6146c9-7f2c-43a4-8e60-3e995a078646" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 11 18:10:33.662670 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:33.662627    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-j7jrb"
Sep 11 18:10:33.663201 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:33.662773    2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 18:10:33.663201 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:33.663187    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j7jrb"
Sep 11 18:10:33.905126 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:33.905093    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:33.905306 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:33.905222    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhvvw" podUID="c882403c-4f90-4d58-a269-9b897f0ba2e8"
Sep 11 18:10:34.904172 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:34.904138    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:34.904649 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:34.904186    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:34.904649 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:34.904274    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-44nnd" podUID="6609afb5-5000-42c3-814e-9f9ee86a32d4"
Sep 11 18:10:34.904649 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:34.904391    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-db9kz" podUID="99a9405b-bada-4d50-a3cb-3cf9eb701eee"
Sep 11 18:10:35.679215 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.679168    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99a9405b-bada-4d50-a3cb-3cf9eb701eee-original-pull-secret\") pod \"global-pull-secret-syncer-db9kz\" (UID: \"99a9405b-bada-4d50-a3cb-3cf9eb701eee\") " pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:35.679215 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.679219    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs\") pod \"network-metrics-daemon-44nnd\" (UID: \"6609afb5-5000-42c3-814e-9f9ee86a32d4\") " pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:35.679517 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:35.679339    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:10:35.679517 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:35.679344    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:10:35.679517 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:35.679409    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs podName:6609afb5-5000-42c3-814e-9f9ee86a32d4 nodeName:}" failed. No retries permitted until 2026-09-11 18:11:07.679389818 +0000 UTC m=+66.280750987 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs") pod "network-metrics-daemon-44nnd" (UID: "6609afb5-5000-42c3-814e-9f9ee86a32d4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:10:35.679517 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:35.679427    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99a9405b-bada-4d50-a3cb-3cf9eb701eee-original-pull-secret podName:99a9405b-bada-4d50-a3cb-3cf9eb701eee nodeName:}" failed. No retries permitted until 2026-09-11 18:11:07.679418464 +0000 UTC m=+66.280779605 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99a9405b-bada-4d50-a3cb-3cf9eb701eee-original-pull-secret") pod "global-pull-secret-syncer-db9kz" (UID: "99a9405b-bada-4d50-a3cb-3cf9eb701eee") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:10:35.714992 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.714963    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-111.ec2.internal" event="NodeReady"
Sep 11 18:10:35.715175 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.715116    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 18:10:35.770789 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.770708    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"]
Sep 11 18:10:35.780168 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.780124    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnjj\" (UniqueName: \"kubernetes.io/projected/c882403c-4f90-4d58-a269-9b897f0ba2e8-kube-api-access-nxnjj\") pod \"network-check-target-qhvvw\" (UID: \"c882403c-4f90-4d58-a269-9b897f0ba2e8\") " pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:35.780353 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:35.780294    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:10:35.780353 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:35.780316    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:10:35.780353 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:35.780327    2576 projected.go:196] Error preparing data for projected volume kube-api-access-nxnjj for pod openshift-network-diagnostics/network-check-target-qhvvw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:10:35.780544 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:35.780380    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c882403c-4f90-4d58-a269-9b897f0ba2e8-kube-api-access-nxnjj podName:c882403c-4f90-4d58-a269-9b897f0ba2e8 nodeName:}" failed. No retries permitted until 2026-09-11 18:11:07.780363185 +0000 UTC m=+66.381724345 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxnjj" (UniqueName: "kubernetes.io/projected/c882403c-4f90-4d58-a269-9b897f0ba2e8-kube-api-access-nxnjj") pod "network-check-target-qhvvw" (UID: "c882403c-4f90-4d58-a269-9b897f0ba2e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:10:35.793902 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.793878    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd46d7d-89rm6"]
Sep 11 18:10:35.794052 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.794032    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:35.796824 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.796606    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ll578\""
Sep 11 18:10:35.796824 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.796772    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 18:10:35.802531 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.802506    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 18:10:35.802766 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.802747    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 18:10:35.810100 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.810079    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74f587c8c6-nkm8p"]
Sep 11 18:10:35.810220 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.810178    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd46d7d-89rm6"
Sep 11 18:10:35.813030 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.813006    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 18:10:35.813575 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.813556    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 18:10:35.813575 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.813565    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 18:10:35.813738 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.813608    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 11 18:10:35.814478 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.814456    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 18:10:35.826714 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.826691    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5"]
Sep 11 18:10:35.826835 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.826819    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74f587c8c6-nkm8p"
Sep 11 18:10:35.831016 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.830995    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 11 18:10:35.831136 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.831120    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-nh7bm\""
Sep 11 18:10:35.836651 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.836628    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xzgnx"]
Sep 11 18:10:35.836758 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.836739    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5"
Sep 11 18:10:35.843912 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.843578    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 11 18:10:35.843912 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.843577    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 11 18:10:35.843912 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.843704    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 11 18:10:35.843912 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.843709    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 11 18:10:35.848986 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.848966    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"]
Sep 11 18:10:35.849113 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.849004    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd46d7d-89rm6"]
Sep 11 18:10:35.849113 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.849029    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74f587c8c6-nkm8p"]
Sep 11 18:10:35.849113 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.849045    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5"]
Sep 11 18:10:35.849113 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.849049    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xzgnx"
Sep 11 18:10:35.849113 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.849062    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xzgnx"]
Sep 11 18:10:35.860335 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.860315    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-p7bc6\""
Sep 11 18:10:35.860445 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.860317    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 18:10:35.860971 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.860950    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 18:10:35.861236 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.861219    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 18:10:35.880735 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.880710    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9add6aa-d540-4a0b-a893-e2e535c266ad-image-registry-private-configuration\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:35.880852 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.880739    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:35.880852 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.880762    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9add6aa-d540-4a0b-a893-e2e535c266ad-ca-trust-extracted\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:35.880852 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.880786    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9add6aa-d540-4a0b-a893-e2e535c266ad-trusted-ca\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:35.880852 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.880811    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkbsq\" (UniqueName: \"kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-kube-api-access-hkbsq\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:35.880852 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.880827    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-certificates\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:35.881094 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.880904    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9add6aa-d540-4a0b-a893-e2e535c266ad-installation-pull-secrets\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:35.881094 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.880969    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-bound-sa-token\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:35.892863 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.892839    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7d7v2"]
Sep 11 18:10:35.908031 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.908001    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:10:35.908415 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.908002    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:10:35.912784 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.912684    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 18:10:35.912784 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.912732    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 18:10:35.912784 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.912754    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4zn8w\""
Sep 11 18:10:35.913007 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.912805    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ztbbx\""
Sep 11 18:10:35.913007 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.912754    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 18:10:35.913007 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.912881    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 18:10:35.925424 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.925370    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7d7v2"]
Sep 11 18:10:35.982058 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.982028    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lftlp\" (UniqueName: \"kubernetes.io/projected/e64c4597-e3bc-46f1-8118-ab47d1e632cb-kube-api-access-lftlp\") pod \"ingress-canary-xzgnx\" (UID: \"e64c4597-e3bc-46f1-8118-ab47d1e632cb\") " pod="openshift-ingress-canary/ingress-canary-xzgnx"
Sep 11 18:10:35.982058 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.982066    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9add6aa-d540-4a0b-a893-e2e535c266ad-installation-pull-secrets\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:35.982309 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.982099    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-bound-sa-token\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:35.982309 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.982170    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlxmx\" (UniqueName: \"kubernetes.io/projected/1e135c2f-cd44-47a0-9478-8e9f6008f629-kube-api-access-vlxmx\") pod \"managed-serviceaccount-addon-agent-74f587c8c6-nkm8p\" (UID: \"1e135c2f-cd44-47a0-9478-8e9f6008f629\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74f587c8c6-nkm8p"
Sep 11 18:10:35.982309 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.982206    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1570f52c-8164-4826-9168-30980530ceeb-ca\") pod \"cluster-proxy-proxy-agent-c86957f4c-99xh5\" (UID: \"1570f52c-8164-4826-9168-30980530ceeb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5"
Sep 11 18:10:35.982309 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.982275    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/359ae04a-a6ec-4689-924d-12199840c2ee-tmp\") pod \"klusterlet-addon-workmgr-654bd46d7d-89rm6\" (UID: \"359ae04a-a6ec-4689-924d-12199840c2ee\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd46d7d-89rm6"
Sep 11 18:10:35.982309 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.982300    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4jgb\" (UniqueName: \"kubernetes.io/projected/359ae04a-a6ec-4689-924d-12199840c2ee-kube-api-access-t4jgb\") pod \"klusterlet-addon-workmgr-654bd46d7d-89rm6\" (UID: \"359ae04a-a6ec-4689-924d-12199840c2ee\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd46d7d-89rm6"
Sep 11 18:10:35.982571 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.982381    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9add6aa-d540-4a0b-a893-e2e535c266ad-image-registry-private-configuration\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:35.982571 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.982418    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:35.982571 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.982446    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9add6aa-d540-4a0b-a893-e2e535c266ad-ca-trust-extracted\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:35.982571 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.982471    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9add6aa-d540-4a0b-a893-e2e535c266ad-trusted-ca\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:35.982571 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.982503    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkbsq\" (UniqueName: \"kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-kube-api-access-hkbsq\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:35.982571 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.982529    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1e135c2f-cd44-47a0-9478-8e9f6008f629-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74f587c8c6-nkm8p\" (UID: \"1e135c2f-cd44-47a0-9478-8e9f6008f629\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74f587c8c6-nkm8p"
Sep 11 18:10:35.982571 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:35.982551    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:10:35.982571 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.982556    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4x7h\" (UniqueName: \"kubernetes.io/projected/1570f52c-8164-4826-9168-30980530ceeb-kube-api-access-g4x7h\") pod \"cluster-proxy-proxy-agent-c86957f4c-99xh5\" (UID: \"1570f52c-8164-4826-9168-30980530ceeb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5"
Sep 11 18:10:35.982571 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:35.982568    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6: secret "image-registry-tls" not found
Sep 11 18:10:35.983016 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.982597    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-certificates\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:35.983016 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.982625    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1570f52c-8164-4826-9168-30980530ceeb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-c86957f4c-99xh5\" (UID: \"1570f52c-8164-4826-9168-30980530ceeb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5"
Sep 11 18:10:35.983016 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:35.982645    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls podName:c9add6aa-d540-4a0b-a893-e2e535c266ad nodeName:}" failed. No retries permitted until 2026-09-11 18:10:36.482624385 +0000 UTC m=+35.083985539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls") pod "image-registry-6c8b6c9dbc-jq7q6" (UID: "c9add6aa-d540-4a0b-a893-e2e535c266ad") : secret "image-registry-tls" not found
Sep 11 18:10:35.983188 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.983054    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9add6aa-d540-4a0b-a893-e2e535c266ad-ca-trust-extracted\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:35.983188 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.983105    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e64c4597-e3bc-46f1-8118-ab47d1e632cb-cert\") pod \"ingress-canary-xzgnx\" (UID: \"e64c4597-e3bc-46f1-8118-ab47d1e632cb\") " pod="openshift-ingress-canary/ingress-canary-xzgnx"
Sep 11 18:10:35.983188 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.983141    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1570f52c-8164-4826-9168-30980530ceeb-hub\") pod \"cluster-proxy-proxy-agent-c86957f4c-99xh5\" (UID: \"1570f52c-8164-4826-9168-30980530ceeb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5"
Sep 11 18:10:35.983188 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.983176    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1570f52c-8164-4826-9168-30980530ceeb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-c86957f4c-99xh5\" (UID: \"1570f52c-8164-4826-9168-30980530ceeb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5"
Sep 11 18:10:35.983448 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.983205    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1570f52c-8164-4826-9168-30980530ceeb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-c86957f4c-99xh5\" (UID: \"1570f52c-8164-4826-9168-30980530ceeb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5"
Sep 11 18:10:35.983448 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.983235    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/359ae04a-a6ec-4689-924d-12199840c2ee-klusterlet-config\") pod \"klusterlet-addon-workmgr-654bd46d7d-89rm6\" (UID: \"359ae04a-a6ec-4689-924d-12199840c2ee\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd46d7d-89rm6"
Sep 11 18:10:35.983555 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.983507    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-certificates\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:35.983729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.983709    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9add6aa-d540-4a0b-a893-e2e535c266ad-trusted-ca\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:35.987345 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.987318    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9add6aa-d540-4a0b-a893-e2e535c266ad-installation-pull-secrets\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:35.987345 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.987335    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9add6aa-d540-4a0b-a893-e2e535c266ad-image-registry-private-configuration\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:35.994628 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.994606    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-bound-sa-token\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:35.994824 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:35.994744    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkbsq\" (UniqueName: \"kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-kube-api-access-hkbsq\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:36.069928 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.069659    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j7jrb_f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232/konnectivity-agent/0.log" limit=1024
Sep 11 18:10:36.083564 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.083533    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1e135c2f-cd44-47a0-9478-8e9f6008f629-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74f587c8c6-nkm8p\" (UID: \"1e135c2f-cd44-47a0-9478-8e9f6008f629\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74f587c8c6-nkm8p"
Sep 11 18:10:36.083564 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.083568    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4x7h\" (UniqueName: \"kubernetes.io/projected/1570f52c-8164-4826-9168-30980530ceeb-kube-api-access-g4x7h\") pod \"cluster-proxy-proxy-agent-c86957f4c-99xh5\" (UID: \"1570f52c-8164-4826-9168-30980530ceeb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5"
Sep 11 18:10:36.083820 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.083599    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1570f52c-8164-4826-9168-30980530ceeb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-c86957f4c-99xh5\" (UID: \"1570f52c-8164-4826-9168-30980530ceeb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5"
Sep 11 18:10:36.083820 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.083617    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e64c4597-e3bc-46f1-8118-ab47d1e632cb-cert\") pod \"ingress-canary-xzgnx\" (UID: \"e64c4597-e3bc-46f1-8118-ab47d1e632cb\") " pod="openshift-ingress-canary/ingress-canary-xzgnx"
Sep 11 18:10:36.083820 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.083738    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1570f52c-8164-4826-9168-30980530ceeb-hub\") pod \"cluster-proxy-proxy-agent-c86957f4c-99xh5\" (UID: \"1570f52c-8164-4826-9168-30980530ceeb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5"
Sep 11 18:10:36.083820 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.083783    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1570f52c-8164-4826-9168-30980530ceeb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-c86957f4c-99xh5\" (UID: \"1570f52c-8164-4826-9168-30980530ceeb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5"
Sep 11 18:10:36.083820 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.083814    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1570f52c-8164-4826-9168-30980530ceeb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-c86957f4c-99xh5\" (UID: \"1570f52c-8164-4826-9168-30980530ceeb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5"
Sep 11 18:10:36.084071 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.083844    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/359ae04a-a6ec-4689-924d-12199840c2ee-klusterlet-config\") pod \"klusterlet-addon-workmgr-654bd46d7d-89rm6\" (UID: \"359ae04a-a6ec-4689-924d-12199840c2ee\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd46d7d-89rm6"
Sep 11 18:10:36.084071 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.083873    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-config-volume\") pod \"dns-default-7d7v2\" (UID: \"b5b1dd54-6a88-46eb-8f13-0e7a54c9490e\") " pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:10:36.084071 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.083903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lftlp\" (UniqueName: \"kubernetes.io/projected/e64c4597-e3bc-46f1-8118-ab47d1e632cb-kube-api-access-lftlp\") pod \"ingress-canary-xzgnx\" (UID: \"e64c4597-e3bc-46f1-8118-ab47d1e632cb\") " pod="openshift-ingress-canary/ingress-canary-xzgnx"
Sep 11 18:10:36.084071 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.083923    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls\") pod \"dns-default-7d7v2\" (UID: \"b5b1dd54-6a88-46eb-8f13-0e7a54c9490e\") " pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:10:36.084071 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.083944    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-tmp-dir\") pod \"dns-default-7d7v2\" (UID: \"b5b1dd54-6a88-46eb-8f13-0e7a54c9490e\") " pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:10:36.084071 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.083968    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5dm9\" (UniqueName: \"kubernetes.io/projected/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-kube-api-access-l5dm9\") pod \"dns-default-7d7v2\" (UID: \"b5b1dd54-6a88-46eb-8f13-0e7a54c9490e\") " pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:10:36.084071 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.084038    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlxmx\" (UniqueName: \"kubernetes.io/projected/1e135c2f-cd44-47a0-9478-8e9f6008f629-kube-api-access-vlxmx\") pod \"managed-serviceaccount-addon-agent-74f587c8c6-nkm8p\" (UID: \"1e135c2f-cd44-47a0-9478-8e9f6008f629\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74f587c8c6-nkm8p"
Sep 11 18:10:36.084071 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.084072    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1570f52c-8164-4826-9168-30980530ceeb-ca\") pod \"cluster-proxy-proxy-agent-c86957f4c-99xh5\" (UID: \"1570f52c-8164-4826-9168-30980530ceeb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5"
Sep 11 18:10:36.084499 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.084118    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/359ae04a-a6ec-4689-924d-12199840c2ee-tmp\") pod \"klusterlet-addon-workmgr-654bd46d7d-89rm6\" (UID: \"359ae04a-a6ec-4689-924d-12199840c2ee\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd46d7d-89rm6"
Sep 11 18:10:36.084499 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.084142    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4jgb\" (UniqueName: \"kubernetes.io/projected/359ae04a-a6ec-4689-924d-12199840c2ee-kube-api-access-t4jgb\") pod \"klusterlet-addon-workmgr-654bd46d7d-89rm6\" (UID: \"359ae04a-a6ec-4689-924d-12199840c2ee\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd46d7d-89rm6"
Sep 11 18:10:36.084499 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.084379    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1570f52c-8164-4826-9168-30980530ceeb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-c86957f4c-99xh5\" (UID: \"1570f52c-8164-4826-9168-30980530ceeb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5"
Sep 11 18:10:36.084858 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.084827    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/359ae04a-a6ec-4689-924d-12199840c2ee-tmp\") pod \"klusterlet-addon-workmgr-654bd46d7d-89rm6\" (UID: \"359ae04a-a6ec-4689-924d-12199840c2ee\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd46d7d-89rm6"
Sep 11 18:10:36.087471 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.087448    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e64c4597-e3bc-46f1-8118-ab47d1e632cb-cert\") pod \"ingress-canary-xzgnx\" (UID: \"e64c4597-e3bc-46f1-8118-ab47d1e632cb\") " pod="openshift-ingress-canary/ingress-canary-xzgnx"
Sep 11 18:10:36.087607 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.087516    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1570f52c-8164-4826-9168-30980530ceeb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-c86957f4c-99xh5\" (UID: \"1570f52c-8164-4826-9168-30980530ceeb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5"
Sep 11 18:10:36.087607 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.087598    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1e135c2f-cd44-47a0-9478-8e9f6008f629-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74f587c8c6-nkm8p\" (UID: \"1e135c2f-cd44-47a0-9478-8e9f6008f629\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74f587c8c6-nkm8p"
Sep 11 18:10:36.087741 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.087611    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/359ae04a-a6ec-4689-924d-12199840c2ee-klusterlet-config\") pod \"klusterlet-addon-workmgr-654bd46d7d-89rm6\" (UID: \"359ae04a-a6ec-4689-924d-12199840c2ee\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd46d7d-89rm6"
Sep 11 18:10:36.087922 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.087899    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1570f52c-8164-4826-9168-30980530ceeb-ca\") pod \"cluster-proxy-proxy-agent-c86957f4c-99xh5\" (UID: \"1570f52c-8164-4826-9168-30980530ceeb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5"
Sep 11 18:10:36.088073 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.088044    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1570f52c-8164-4826-9168-30980530ceeb-hub\") pod \"cluster-proxy-proxy-agent-c86957f4c-99xh5\" (UID: \"1570f52c-8164-4826-9168-30980530ceeb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5"
Sep 11 18:10:36.088688 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.088667    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1570f52c-8164-4826-9168-30980530ceeb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-c86957f4c-99xh5\" (UID: \"1570f52c-8164-4826-9168-30980530ceeb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5"
Sep 11 18:10:36.092991 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.092964    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lftlp\" (UniqueName: \"kubernetes.io/projected/e64c4597-e3bc-46f1-8118-ab47d1e632cb-kube-api-access-lftlp\") pod \"ingress-canary-xzgnx\" (UID: \"e64c4597-e3bc-46f1-8118-ab47d1e632cb\") " pod="openshift-ingress-canary/ingress-canary-xzgnx"
Sep 11 18:10:36.093397 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.093375    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlxmx\" (UniqueName: \"kubernetes.io/projected/1e135c2f-cd44-47a0-9478-8e9f6008f629-kube-api-access-vlxmx\") pod \"managed-serviceaccount-addon-agent-74f587c8c6-nkm8p\" (UID: \"1e135c2f-cd44-47a0-9478-8e9f6008f629\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74f587c8c6-nkm8p"
Sep 11 18:10:36.093527 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.093437    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4jgb\" (UniqueName: \"kubernetes.io/projected/359ae04a-a6ec-4689-924d-12199840c2ee-kube-api-access-t4jgb\") pod \"klusterlet-addon-workmgr-654bd46d7d-89rm6\" (UID: \"359ae04a-a6ec-4689-924d-12199840c2ee\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd46d7d-89rm6"
Sep 11 18:10:36.093632 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.093612    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4x7h\" (UniqueName: \"kubernetes.io/projected/1570f52c-8164-4826-9168-30980530ceeb-kube-api-access-g4x7h\") pod \"cluster-proxy-proxy-agent-c86957f4c-99xh5\" (UID: \"1570f52c-8164-4826-9168-30980530ceeb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5"
Sep 11 18:10:36.120647 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.120614    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd46d7d-89rm6"
Sep 11 18:10:36.145970 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.145932    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74f587c8c6-nkm8p"
Sep 11 18:10:36.156965 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.156943    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5"
Sep 11 18:10:36.171340 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.171318    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xzgnx"
Sep 11 18:10:36.180747 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.180723    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j7jrb_f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232/konnectivity-agent/0.log" limit=1024
Sep 11 18:10:36.184524 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.184498    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-config-volume\") pod \"dns-default-7d7v2\" (UID: \"b5b1dd54-6a88-46eb-8f13-0e7a54c9490e\") " pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:10:36.184655 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.184538    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls\") pod \"dns-default-7d7v2\" (UID: \"b5b1dd54-6a88-46eb-8f13-0e7a54c9490e\") " pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:10:36.184655 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.184562    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-tmp-dir\") pod \"dns-default-7d7v2\" (UID: \"b5b1dd54-6a88-46eb-8f13-0e7a54c9490e\") " pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:10:36.184655 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.184578    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5dm9\" (UniqueName: \"kubernetes.io/projected/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-kube-api-access-l5dm9\") pod \"dns-default-7d7v2\" (UID: \"b5b1dd54-6a88-46eb-8f13-0e7a54c9490e\") " pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:10:36.184801 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:36.184705    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:10:36.184801 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:36.184778    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls podName:b5b1dd54-6a88-46eb-8f13-0e7a54c9490e nodeName:}" failed. No retries permitted until 2026-09-11 18:10:36.684757053 +0000 UTC m=+35.286118210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls") pod "dns-default-7d7v2" (UID: "b5b1dd54-6a88-46eb-8f13-0e7a54c9490e") : secret "dns-default-metrics-tls" not found
Sep 11 18:10:36.184945 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.184904    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-tmp-dir\") pod \"dns-default-7d7v2\" (UID: \"b5b1dd54-6a88-46eb-8f13-0e7a54c9490e\") " pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:10:36.188601 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.188564    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-config-volume\") pod \"dns-default-7d7v2\" (UID: \"b5b1dd54-6a88-46eb-8f13-0e7a54c9490e\") " pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:10:36.193793 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.193768    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5dm9\" (UniqueName: \"kubernetes.io/projected/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-kube-api-access-l5dm9\") pod \"dns-default-7d7v2\" (UID: \"b5b1dd54-6a88-46eb-8f13-0e7a54c9490e\") " pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:10:36.487691 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.487652    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:36.487961 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:36.487791    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:10:36.487961 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:36.487808    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6: secret "image-registry-tls" not found
Sep 11 18:10:36.487961 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:36.487867    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls podName:c9add6aa-d540-4a0b-a893-e2e535c266ad nodeName:}" failed. No retries permitted until 2026-09-11 18:10:37.487847802 +0000 UTC m=+36.089208962 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls") pod "image-registry-6c8b6c9dbc-jq7q6" (UID: "c9add6aa-d540-4a0b-a893-e2e535c266ad") : secret "image-registry-tls" not found
Sep 11 18:10:36.690150 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.690118    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls\") pod \"dns-default-7d7v2\" (UID: \"b5b1dd54-6a88-46eb-8f13-0e7a54c9490e\") " pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:10:36.690325 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:36.690296    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:10:36.690388 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:36.690376    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls podName:b5b1dd54-6a88-46eb-8f13-0e7a54c9490e nodeName:}" failed. No retries permitted until 2026-09-11 18:10:37.690354612 +0000 UTC m=+36.291715754 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls") pod "dns-default-7d7v2" (UID: "b5b1dd54-6a88-46eb-8f13-0e7a54c9490e") : secret "dns-default-metrics-tls" not found
Sep 11 18:10:36.904956 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.904924    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:10:36.905103 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.905086    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:10:36.907521 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.907498    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 18:10:36.907652 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.907553    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bvnsl\""
Sep 11 18:10:36.907652 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:36.907642    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 18:10:37.079229 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:37.079195    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74f587c8c6-nkm8p"]
Sep 11 18:10:37.082121 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:37.082080    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e135c2f_cd44_47a0_9478_8e9f6008f629.slice/crio-3dc0a4d6df965c7a5ab7d73206f17a108cde33c343555385332e7c249d306108 WatchSource:0}: Error finding container 3dc0a4d6df965c7a5ab7d73206f17a108cde33c343555385332e7c249d306108: Status 404 returned error can't find the container with id 3dc0a4d6df965c7a5ab7d73206f17a108cde33c343555385332e7c249d306108
Sep 11 18:10:37.089833 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:37.089388    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xzgnx"]
Sep 11 18:10:37.091178 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:37.091043    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5"]
Sep 11 18:10:37.092136 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:37.092109    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd46d7d-89rm6"]
Sep 11 18:10:37.093286 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:37.093255    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359ae04a_a6ec_4689_924d_12199840c2ee.slice/crio-7174de6ec7aa823828b93c84f4663ac091a53ddaa22f202715cf375262140110 WatchSource:0}: Error finding container 7174de6ec7aa823828b93c84f4663ac091a53ddaa22f202715cf375262140110: Status 404 returned error can't find the container with id 7174de6ec7aa823828b93c84f4663ac091a53ddaa22f202715cf375262140110
Sep 11 18:10:37.093535 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:37.093509    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1570f52c_8164_4826_9168_30980530ceeb.slice/crio-e3d33f1aeee6473162746682351c1e82b254326aaed41f680c89678d74694b39 WatchSource:0}: Error finding container e3d33f1aeee6473162746682351c1e82b254326aaed41f680c89678d74694b39: Status 404 returned error can't find the container with id e3d33f1aeee6473162746682351c1e82b254326aaed41f680c89678d74694b39
Sep 11 18:10:37.094538 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:10:37.094518    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64c4597_e3bc_46f1_8118_ab47d1e632cb.slice/crio-9af7e41ef3e967daa5e4230ed7604f66a3bf69863ce4d830293c2dd15991bde0 WatchSource:0}: Error finding container 9af7e41ef3e967daa5e4230ed7604f66a3bf69863ce4d830293c2dd15991bde0: Status 404 returned error can't find the container with id 9af7e41ef3e967daa5e4230ed7604f66a3bf69863ce4d830293c2dd15991bde0
Sep 11 18:10:37.150079 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:37.150044    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-759db" event={"ID":"fe7af037-cf3e-4ae8-bdf0-f458a249950f","Type":"ContainerStarted","Data":"f160fc6dd79ff017e476ed8ea1d78b08ca8094a3a8798995893eb8322c2cd278"}
Sep 11 18:10:37.151307 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:37.151267    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5" event={"ID":"1570f52c-8164-4826-9168-30980530ceeb","Type":"ContainerStarted","Data":"e3d33f1aeee6473162746682351c1e82b254326aaed41f680c89678d74694b39"}
Sep 11 18:10:37.152424 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:37.152398    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd46d7d-89rm6" event={"ID":"359ae04a-a6ec-4689-924d-12199840c2ee","Type":"ContainerStarted","Data":"7174de6ec7aa823828b93c84f4663ac091a53ddaa22f202715cf375262140110"}
Sep 11 18:10:37.153556 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:37.153534    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74f587c8c6-nkm8p" event={"ID":"1e135c2f-cd44-47a0-9478-8e9f6008f629","Type":"ContainerStarted","Data":"3dc0a4d6df965c7a5ab7d73206f17a108cde33c343555385332e7c249d306108"}
Sep 11 18:10:37.154785 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:37.154764    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xzgnx" event={"ID":"e64c4597-e3bc-46f1-8118-ab47d1e632cb","Type":"ContainerStarted","Data":"9af7e41ef3e967daa5e4230ed7604f66a3bf69863ce4d830293c2dd15991bde0"}
Sep 11 18:10:37.498958 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:37.498926    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:37.499170 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:37.499057    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:10:37.499170 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:37.499073    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6: secret "image-registry-tls" not found
Sep 11 18:10:37.499170 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:37.499135    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls podName:c9add6aa-d540-4a0b-a893-e2e535c266ad nodeName:}" failed. No retries permitted until 2026-09-11 18:10:39.499117476 +0000 UTC m=+38.100478625 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls") pod "image-registry-6c8b6c9dbc-jq7q6" (UID: "c9add6aa-d540-4a0b-a893-e2e535c266ad") : secret "image-registry-tls" not found
Sep 11 18:10:37.700574 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:37.700544    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls\") pod \"dns-default-7d7v2\" (UID: \"b5b1dd54-6a88-46eb-8f13-0e7a54c9490e\") " pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:10:37.700753 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:37.700698    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:10:37.700800 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:37.700756    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls podName:b5b1dd54-6a88-46eb-8f13-0e7a54c9490e nodeName:}" failed. No retries permitted until 2026-09-11 18:10:39.700741526 +0000 UTC m=+38.302102687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls") pod "dns-default-7d7v2" (UID: "b5b1dd54-6a88-46eb-8f13-0e7a54c9490e") : secret "dns-default-metrics-tls" not found
Sep 11 18:10:38.163567 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:38.163525    2576 generic.go:356] "Generic (PLEG): container finished" podID="fe7af037-cf3e-4ae8-bdf0-f458a249950f" containerID="f160fc6dd79ff017e476ed8ea1d78b08ca8094a3a8798995893eb8322c2cd278" exitCode=0
Sep 11 18:10:38.164191 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:38.163603    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-759db" event={"ID":"fe7af037-cf3e-4ae8-bdf0-f458a249950f","Type":"ContainerDied","Data":"f160fc6dd79ff017e476ed8ea1d78b08ca8094a3a8798995893eb8322c2cd278"}
Sep 11 18:10:39.174237 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:39.174200    2576 generic.go:356] "Generic (PLEG): container finished" podID="fe7af037-cf3e-4ae8-bdf0-f458a249950f" containerID="5935f94d1e27d102cb1b45a8ff3109c7bf1bf8a73fa15f3bf052abff7b465cd6" exitCode=0
Sep 11 18:10:39.175896 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:39.174258    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-759db" event={"ID":"fe7af037-cf3e-4ae8-bdf0-f458a249950f","Type":"ContainerDied","Data":"5935f94d1e27d102cb1b45a8ff3109c7bf1bf8a73fa15f3bf052abff7b465cd6"}
Sep 11 18:10:39.523073 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:39.522819    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:39.523308 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:39.523288    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:10:39.523382 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:39.523313    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6: secret "image-registry-tls" not found
Sep 11 18:10:39.523382 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:39.523371    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls podName:c9add6aa-d540-4a0b-a893-e2e535c266ad nodeName:}" failed. No retries permitted until 2026-09-11 18:10:43.523352289 +0000 UTC m=+42.124713450 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls") pod "image-registry-6c8b6c9dbc-jq7q6" (UID: "c9add6aa-d540-4a0b-a893-e2e535c266ad") : secret "image-registry-tls" not found
Sep 11 18:10:39.724148 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:39.724106    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls\") pod \"dns-default-7d7v2\" (UID: \"b5b1dd54-6a88-46eb-8f13-0e7a54c9490e\") " pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:10:39.724393 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:39.724363    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:10:39.724473 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:39.724428    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls podName:b5b1dd54-6a88-46eb-8f13-0e7a54c9490e nodeName:}" failed. No retries permitted until 2026-09-11 18:10:43.724410157 +0000 UTC m=+42.325771304 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls") pod "dns-default-7d7v2" (UID: "b5b1dd54-6a88-46eb-8f13-0e7a54c9490e") : secret "dns-default-metrics-tls" not found
Sep 11 18:10:40.182197 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:40.182154    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-759db" event={"ID":"fe7af037-cf3e-4ae8-bdf0-f458a249950f","Type":"ContainerStarted","Data":"c3eb6868c028780c127011c77c6092f101c422bbdc79923762b2be7925321d78"}
Sep 11 18:10:40.215496 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:40.215429    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-759db" podStartSLOduration=9.157887012 podStartE2EDuration="38.215407327s" totalImagesPullingTime="29.057520315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:10:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:10:40.214574519 +0000 UTC m=+38.815935677" watchObservedRunningTime="2026-09-11 18:10:40.215407327 +0000 UTC m=+38.816768487"
Sep 11 18:10:40.339960 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:40.339919    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j7jrb_f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232/konnectivity-agent/0.log" limit=1024
Sep 11 18:10:40.408069 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:40.408027    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j7jrb_f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232/konnectivity-agent/0.log" limit=1024
Sep 11 18:10:41.213499 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:41.213461    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j7jrb_f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232/konnectivity-agent/0.log" limit=1024
Sep 11 18:10:41.314508 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:41.314461    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j7jrb_f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232/konnectivity-agent/0.log" limit=1024
Sep 11 18:10:41.414126 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:41.414054    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j7jrb_f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232/konnectivity-agent/0.log" limit=1024
Sep 11 18:10:41.514916 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:41.514796    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j7jrb_f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232/konnectivity-agent/0.log" limit=1024
Sep 11 18:10:43.246750 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:43.246708    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j7jrb_f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232/konnectivity-agent/0.log" limit=1024
Sep 11 18:10:43.351489 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:43.351454    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j7jrb_f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232/konnectivity-agent/0.log" limit=1024
Sep 11 18:10:43.451846 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:43.451812    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j7jrb_f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232/konnectivity-agent/0.log" limit=1024
Sep 11 18:10:43.554222 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:43.554142    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:43.554391 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:43.554292    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:10:43.554391 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:43.554306    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6: secret "image-registry-tls" not found
Sep 11 18:10:43.554391 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:43.554363    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls podName:c9add6aa-d540-4a0b-a893-e2e535c266ad nodeName:}" failed. No retries permitted until 2026-09-11 18:10:51.554344385 +0000 UTC m=+50.155705531 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls") pod "image-registry-6c8b6c9dbc-jq7q6" (UID: "c9add6aa-d540-4a0b-a893-e2e535c266ad") : secret "image-registry-tls" not found
Sep 11 18:10:43.755308 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:43.755272    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls\") pod \"dns-default-7d7v2\" (UID: \"b5b1dd54-6a88-46eb-8f13-0e7a54c9490e\") " pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:10:43.755492 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:43.755392    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:10:43.755492 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:43.755456    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls podName:b5b1dd54-6a88-46eb-8f13-0e7a54c9490e nodeName:}" failed. No retries permitted until 2026-09-11 18:10:51.755436094 +0000 UTC m=+50.356797258 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls") pod "dns-default-7d7v2" (UID: "b5b1dd54-6a88-46eb-8f13-0e7a54c9490e") : secret "dns-default-metrics-tls" not found
Sep 11 18:10:46.200807 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:46.200771    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5" event={"ID":"1570f52c-8164-4826-9168-30980530ceeb","Type":"ContainerStarted","Data":"40fe07540bc5093bf204340b8377080e4b8000d10082f444970764938d7bc1bb"}
Sep 11 18:10:46.201250 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:46.200824    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5" event={"ID":"1570f52c-8164-4826-9168-30980530ceeb","Type":"ContainerStarted","Data":"fe699b90a6b1289dde09e5f9947bd83b4d51d4d486f9bfa109cf2e2f9957588c"}
Sep 11 18:10:46.202898 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:46.202615    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd46d7d-89rm6" event={"ID":"359ae04a-a6ec-4689-924d-12199840c2ee","Type":"ContainerStarted","Data":"35669fdea07d2401fb880d37688a65c0d7640eb44d68420577980829a324c022"}
Sep 11 18:10:46.203108 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:46.203062    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd46d7d-89rm6"
Sep 11 18:10:46.204719 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:46.204697    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd46d7d-89rm6"
Sep 11 18:10:46.204955 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:46.204829    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74f587c8c6-nkm8p" event={"ID":"1e135c2f-cd44-47a0-9478-8e9f6008f629","Type":"ContainerStarted","Data":"c23430776b11230caa1c9905705fc21d70295e93b3b86b54875f59162ccaf209"}
Sep 11 18:10:46.206527 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:46.206506    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xzgnx" event={"ID":"e64c4597-e3bc-46f1-8118-ab47d1e632cb","Type":"ContainerStarted","Data":"6a4692cbcf29a4fc7542bf0d77f5367c0a813713fb0c87e2cb681a33e4eb27f4"}
Sep 11 18:10:46.231116 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:46.228357    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd46d7d-89rm6" podStartSLOduration=27.419535326 podStartE2EDuration="36.228337447s" totalImagesPullingTime="8.808802121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:10:46.227829796 +0000 UTC m=+44.829190963" watchObservedRunningTime="2026-09-11 18:10:46.228337447 +0000 UTC m=+44.829698612"
Sep 11 18:10:46.275206 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:46.275157    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74f587c8c6-nkm8p" podStartSLOduration=27.471787018 podStartE2EDuration="36.275140971s" totalImagesPullingTime="8.803353953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:10:46.274758615 +0000 UTC m=+44.876119790" watchObservedRunningTime="2026-09-11 18:10:46.275140971 +0000 UTC m=+44.876502133"
Sep 11 18:10:46.297230 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:46.297179    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xzgnx" podStartSLOduration=2.5076912460000003 podStartE2EDuration="11.297162915s" totalImagesPullingTime="8.789471669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:10:46.296559623 +0000 UTC m=+44.897920787" watchObservedRunningTime="2026-09-11 18:10:46.297162915 +0000 UTC m=+44.898524071"
Sep 11 18:10:47.211614 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:47.211565    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5" event={"ID":"1570f52c-8164-4826-9168-30980530ceeb","Type":"ContainerStarted","Data":"1bf0778574efac507af54491361b033010a457aa2982b3f8af8bc19675109097"}
Sep 11 18:10:47.235841 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:47.235788    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5" podStartSLOduration=28.442598592 podStartE2EDuration="37.235772551s" totalImagesPullingTime="8.793173959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:10:47.234095896 +0000 UTC m=+45.835457060" watchObservedRunningTime="2026-09-11 18:10:47.235772551 +0000 UTC m=+45.837133732"
Sep 11 18:10:51.611745 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:51.611704    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:10:51.612166 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:51.611859    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:10:51.612166 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:51.611881    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6: secret "image-registry-tls" not found
Sep 11 18:10:51.612166 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:51.611938    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls podName:c9add6aa-d540-4a0b-a893-e2e535c266ad nodeName:}" failed. No retries permitted until 2026-09-11 18:11:07.611921739 +0000 UTC m=+66.213282885 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls") pod "image-registry-6c8b6c9dbc-jq7q6" (UID: "c9add6aa-d540-4a0b-a893-e2e535c266ad") : secret "image-registry-tls" not found
Sep 11 18:10:51.814309 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:10:51.814270    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls\") pod \"dns-default-7d7v2\" (UID: \"b5b1dd54-6a88-46eb-8f13-0e7a54c9490e\") " pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:10:51.814484 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:51.814427    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:10:51.814525 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:10:51.814490    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls podName:b5b1dd54-6a88-46eb-8f13-0e7a54c9490e nodeName:}" failed. No retries permitted until 2026-09-11 18:11:07.814474917 +0000 UTC m=+66.415836082 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls") pod "dns-default-7d7v2" (UID: "b5b1dd54-6a88-46eb-8f13-0e7a54c9490e") : secret "dns-default-metrics-tls" not found
Sep 11 18:11:03.105431 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:03.105400    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rxclq"
Sep 11 18:11:07.638632 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:07.638571    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:11:07.639007 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:07.638728    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:11:07.639007 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:07.638750    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6: secret "image-registry-tls" not found
Sep 11 18:11:07.639007 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:07.638805    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls podName:c9add6aa-d540-4a0b-a893-e2e535c266ad nodeName:}" failed. No retries permitted until 2026-09-11 18:11:39.638789183 +0000 UTC m=+98.240150328 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls") pod "image-registry-6c8b6c9dbc-jq7q6" (UID: "c9add6aa-d540-4a0b-a893-e2e535c266ad") : secret "image-registry-tls" not found
Sep 11 18:11:07.739268 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:07.739229    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99a9405b-bada-4d50-a3cb-3cf9eb701eee-original-pull-secret\") pod \"global-pull-secret-syncer-db9kz\" (UID: \"99a9405b-bada-4d50-a3cb-3cf9eb701eee\") " pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:11:07.739268 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:07.739270    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs\") pod \"network-metrics-daemon-44nnd\" (UID: \"6609afb5-5000-42c3-814e-9f9ee86a32d4\") " pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:11:07.742298 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:07.742277    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 18:11:07.742343 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:07.742303    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 18:11:07.750271 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:07.750247    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 18:11:07.750393 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:07.750347    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs podName:6609afb5-5000-42c3-814e-9f9ee86a32d4 nodeName:}" failed. No retries permitted until 2026-09-11 18:12:11.750329326 +0000 UTC m=+130.351690473 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs") pod "network-metrics-daemon-44nnd" (UID: "6609afb5-5000-42c3-814e-9f9ee86a32d4") : secret "metrics-daemon-secret" not found
Sep 11 18:11:07.753377 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:07.753355    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99a9405b-bada-4d50-a3cb-3cf9eb701eee-original-pull-secret\") pod \"global-pull-secret-syncer-db9kz\" (UID: \"99a9405b-bada-4d50-a3cb-3cf9eb701eee\") " pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:11:07.819975 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:07.819940    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-db9kz"
Sep 11 18:11:07.839916 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:07.839888    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls\") pod \"dns-default-7d7v2\" (UID: \"b5b1dd54-6a88-46eb-8f13-0e7a54c9490e\") " pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:11:07.840026 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:07.839940    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnjj\" (UniqueName: \"kubernetes.io/projected/c882403c-4f90-4d58-a269-9b897f0ba2e8-kube-api-access-nxnjj\") pod \"network-check-target-qhvvw\" (UID: \"c882403c-4f90-4d58-a269-9b897f0ba2e8\") " pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:11:07.840074 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:07.840046    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:11:07.840136 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:07.840124    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls podName:b5b1dd54-6a88-46eb-8f13-0e7a54c9490e nodeName:}" failed. No retries permitted until 2026-09-11 18:11:39.840104728 +0000 UTC m=+98.441465879 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls") pod "dns-default-7d7v2" (UID: "b5b1dd54-6a88-46eb-8f13-0e7a54c9490e") : secret "dns-default-metrics-tls" not found
Sep 11 18:11:07.843092 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:07.843068    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 18:11:07.852833 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:07.852814    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 18:11:07.864209 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:07.863781    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxnjj\" (UniqueName: \"kubernetes.io/projected/c882403c-4f90-4d58-a269-9b897f0ba2e8-kube-api-access-nxnjj\") pod \"network-check-target-qhvvw\" (UID: \"c882403c-4f90-4d58-a269-9b897f0ba2e8\") " pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:11:08.021855 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:08.021826    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ztbbx\""
Sep 11 18:11:08.029373 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:08.029358    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:11:08.093135 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:08.093106    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-db9kz"]
Sep 11 18:11:08.096037 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:11:08.096013    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a9405b_bada_4d50_a3cb_3cf9eb701eee.slice/crio-d096da79a37020f560294816b2519b9909a865f38c5fb4b12a31fc9cb4a3cbee WatchSource:0}: Error finding container d096da79a37020f560294816b2519b9909a865f38c5fb4b12a31fc9cb4a3cbee: Status 404 returned error can't find the container with id d096da79a37020f560294816b2519b9909a865f38c5fb4b12a31fc9cb4a3cbee
Sep 11 18:11:08.275620 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:08.275534    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-db9kz" event={"ID":"99a9405b-bada-4d50-a3cb-3cf9eb701eee","Type":"ContainerStarted","Data":"d096da79a37020f560294816b2519b9909a865f38c5fb4b12a31fc9cb4a3cbee"}
Sep 11 18:11:08.296270 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:08.296244    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qhvvw"]
Sep 11 18:11:08.299085 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:11:08.299063    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc882403c_4f90_4d58_a269_9b897f0ba2e8.slice/crio-2da3d8d09728fef936bb96e2306e56780798262e939e0b8e6a4a8862a9e8e10d WatchSource:0}: Error finding container 2da3d8d09728fef936bb96e2306e56780798262e939e0b8e6a4a8862a9e8e10d: Status 404 returned error can't find the container with id 2da3d8d09728fef936bb96e2306e56780798262e939e0b8e6a4a8862a9e8e10d
Sep 11 18:11:09.280352 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:09.280312    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qhvvw" event={"ID":"c882403c-4f90-4d58-a269-9b897f0ba2e8","Type":"ContainerStarted","Data":"2da3d8d09728fef936bb96e2306e56780798262e939e0b8e6a4a8862a9e8e10d"}
Sep 11 18:11:14.296728 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:14.296685    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-db9kz" event={"ID":"99a9405b-bada-4d50-a3cb-3cf9eb701eee","Type":"ContainerStarted","Data":"d18f6aa54fd9820acfc274c0755b345a9b1161c3825a349a81e8340f740fe975"}
Sep 11 18:11:14.298125 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:14.298100    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qhvvw" event={"ID":"c882403c-4f90-4d58-a269-9b897f0ba2e8","Type":"ContainerStarted","Data":"ac004cd862267db4b313d44c1ba98f5133291ef7a109740d37b7a7ca9c4ac502"}
Sep 11 18:11:14.298252 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:14.298235    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:11:14.312925 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:14.312883    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-db9kz" podStartSLOduration=66.838153513 podStartE2EDuration="1m12.312868559s" totalImagesPullingTime="5.474715046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:11:14.312050044 +0000 UTC m=+72.913411209" watchObservedRunningTime="2026-09-11 18:11:14.312868559 +0000 UTC m=+72.914229747"
Sep 11 18:11:14.327834 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:14.327797    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qhvvw" podStartSLOduration=67.04757551 podStartE2EDuration="1m12.327783842s" totalImagesPullingTime="5.280208332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:11:14.327350886 +0000 UTC m=+72.928712050" watchObservedRunningTime="2026-09-11 18:11:14.327783842 +0000 UTC m=+72.929144999"
Sep 11 18:11:33.841884 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:33.841770    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9zc7s"]
Sep 11 18:11:33.844453 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:33.844420    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:33.846749 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:33.846728    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8hchm\""
Sep 11 18:11:33.847259 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:33.847233    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 18:11:33.847497 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:33.847482    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 18:11:33.847615 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:33.847514    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 18:11:33.847615 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:33.847566    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 18:11:33.859857 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:33.859837    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9zc7s"]
Sep 11 18:11:33.919078 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:33.919053    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7dcfb0b-9065-4c7e-a163-73844899ac4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:33.919222 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:33.919133    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2tdr\" (UniqueName: \"kubernetes.io/projected/f7dcfb0b-9065-4c7e-a163-73844899ac4c-kube-api-access-s2tdr\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:33.919222 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:33.919157    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f7dcfb0b-9065-4c7e-a163-73844899ac4c-crio-socket\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:33.919222 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:33.919171    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7dcfb0b-9065-4c7e-a163-73844899ac4c-data-volume\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:33.919356 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:33.919226    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7dcfb0b-9065-4c7e-a163-73844899ac4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:34.019648 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:34.019609    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7dcfb0b-9065-4c7e-a163-73844899ac4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:34.019648 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:34.019651    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7dcfb0b-9065-4c7e-a163-73844899ac4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:34.019839 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:34.019717    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2tdr\" (UniqueName: \"kubernetes.io/projected/f7dcfb0b-9065-4c7e-a163-73844899ac4c-kube-api-access-s2tdr\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:34.019839 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:34.019736    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f7dcfb0b-9065-4c7e-a163-73844899ac4c-crio-socket\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:34.019839 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:34.019751    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7dcfb0b-9065-4c7e-a163-73844899ac4c-data-volume\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:34.019839 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:34.019762    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:11:34.019839 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:34.019831    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7dcfb0b-9065-4c7e-a163-73844899ac4c-insights-runtime-extractor-tls podName:f7dcfb0b-9065-4c7e-a163-73844899ac4c nodeName:}" failed. No retries permitted until 2026-09-11 18:11:34.519815885 +0000 UTC m=+93.121177026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7dcfb0b-9065-4c7e-a163-73844899ac4c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9zc7s" (UID: "f7dcfb0b-9065-4c7e-a163-73844899ac4c") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:11:34.019839 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:34.019832    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f7dcfb0b-9065-4c7e-a163-73844899ac4c-crio-socket\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:34.020064 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:34.019886    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:11:34.020064 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:34.019967    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7dcfb0b-9065-4c7e-a163-73844899ac4c-kube-rbac-proxy-cm podName:f7dcfb0b-9065-4c7e-a163-73844899ac4c nodeName:}" failed. No retries permitted until 2026-09-11 18:11:34.519950064 +0000 UTC m=+93.121311231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7dcfb0b-9065-4c7e-a163-73844899ac4c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9zc7s" (UID: "f7dcfb0b-9065-4c7e-a163-73844899ac4c") : configmap "kube-rbac-proxy" not found
Sep 11 18:11:34.020064 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:34.020034    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7dcfb0b-9065-4c7e-a163-73844899ac4c-data-volume\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:34.028844 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:34.028819    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2tdr\" (UniqueName: \"kubernetes.io/projected/f7dcfb0b-9065-4c7e-a163-73844899ac4c-kube-api-access-s2tdr\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:34.523059 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:34.523031    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7dcfb0b-9065-4c7e-a163-73844899ac4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:34.523059 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:34.523063    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7dcfb0b-9065-4c7e-a163-73844899ac4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:34.523286 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:34.523168    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:11:34.523286 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:34.523187    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:11:34.523286 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:34.523219    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7dcfb0b-9065-4c7e-a163-73844899ac4c-kube-rbac-proxy-cm podName:f7dcfb0b-9065-4c7e-a163-73844899ac4c nodeName:}" failed. No retries permitted until 2026-09-11 18:11:35.523206536 +0000 UTC m=+94.124567676 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7dcfb0b-9065-4c7e-a163-73844899ac4c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9zc7s" (UID: "f7dcfb0b-9065-4c7e-a163-73844899ac4c") : configmap "kube-rbac-proxy" not found
Sep 11 18:11:34.523286 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:34.523249    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7dcfb0b-9065-4c7e-a163-73844899ac4c-insights-runtime-extractor-tls podName:f7dcfb0b-9065-4c7e-a163-73844899ac4c nodeName:}" failed. No retries permitted until 2026-09-11 18:11:35.523235664 +0000 UTC m=+94.124596823 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7dcfb0b-9065-4c7e-a163-73844899ac4c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9zc7s" (UID: "f7dcfb0b-9065-4c7e-a163-73844899ac4c") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:11:35.530196 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:35.530158    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7dcfb0b-9065-4c7e-a163-73844899ac4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:35.530196 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:35.530199    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7dcfb0b-9065-4c7e-a163-73844899ac4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:35.530678 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:35.530270    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:11:35.530678 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:35.530301    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:11:35.530678 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:35.530317    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7dcfb0b-9065-4c7e-a163-73844899ac4c-kube-rbac-proxy-cm podName:f7dcfb0b-9065-4c7e-a163-73844899ac4c nodeName:}" failed. No retries permitted until 2026-09-11 18:11:37.530303573 +0000 UTC m=+96.131664718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7dcfb0b-9065-4c7e-a163-73844899ac4c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9zc7s" (UID: "f7dcfb0b-9065-4c7e-a163-73844899ac4c") : configmap "kube-rbac-proxy" not found
Sep 11 18:11:35.530678 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:35.530351    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7dcfb0b-9065-4c7e-a163-73844899ac4c-insights-runtime-extractor-tls podName:f7dcfb0b-9065-4c7e-a163-73844899ac4c nodeName:}" failed. No retries permitted until 2026-09-11 18:11:37.530337832 +0000 UTC m=+96.131698977 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7dcfb0b-9065-4c7e-a163-73844899ac4c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9zc7s" (UID: "f7dcfb0b-9065-4c7e-a163-73844899ac4c") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:11:36.391101 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:36.391070    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nqz9g_e0a9600b-c5a4-4e01-99b5-fe0598a4cb52/dns-node-resolver/0.log"
Sep 11 18:11:37.545689 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:37.545634    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7dcfb0b-9065-4c7e-a163-73844899ac4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:37.545689 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:37.545686    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7dcfb0b-9065-4c7e-a163-73844899ac4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:37.546132 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:37.545768    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:11:37.546132 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:37.545792    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:11:37.546132 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:37.545821    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7dcfb0b-9065-4c7e-a163-73844899ac4c-kube-rbac-proxy-cm podName:f7dcfb0b-9065-4c7e-a163-73844899ac4c nodeName:}" failed. No retries permitted until 2026-09-11 18:11:41.545807843 +0000 UTC m=+100.147168988 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7dcfb0b-9065-4c7e-a163-73844899ac4c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9zc7s" (UID: "f7dcfb0b-9065-4c7e-a163-73844899ac4c") : configmap "kube-rbac-proxy" not found
Sep 11 18:11:37.546132 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:37.545846    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7dcfb0b-9065-4c7e-a163-73844899ac4c-insights-runtime-extractor-tls podName:f7dcfb0b-9065-4c7e-a163-73844899ac4c nodeName:}" failed. No retries permitted until 2026-09-11 18:11:41.54583337 +0000 UTC m=+100.147194515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7dcfb0b-9065-4c7e-a163-73844899ac4c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9zc7s" (UID: "f7dcfb0b-9065-4c7e-a163-73844899ac4c") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:11:37.789408 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:37.789372    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ckcv2_9c70edc6-7469-4d7c-b3f1-5d88caac76aa/node-ca/0.log"
Sep 11 18:11:38.589404 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:38.589363    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xzgnx_e64c4597-e3bc-46f1-8118-ab47d1e632cb/serve-healthcheck-canary/0.log"
Sep 11 18:11:39.660078 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:39.660035    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls\") pod \"image-registry-6c8b6c9dbc-jq7q6\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") " pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:11:39.660483 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:39.660189    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:11:39.660483 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:39.660207    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6: secret "image-registry-tls" not found
Sep 11 18:11:39.660483 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:39.660258    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls podName:c9add6aa-d540-4a0b-a893-e2e535c266ad nodeName:}" failed. No retries permitted until 2026-09-11 18:12:43.660243847 +0000 UTC m=+162.261604988 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls") pod "image-registry-6c8b6c9dbc-jq7q6" (UID: "c9add6aa-d540-4a0b-a893-e2e535c266ad") : secret "image-registry-tls" not found
Sep 11 18:11:39.861375 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:39.861343    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls\") pod \"dns-default-7d7v2\" (UID: \"b5b1dd54-6a88-46eb-8f13-0e7a54c9490e\") " pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:11:39.861517 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:39.861462    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:11:39.861517 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:39.861514    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls podName:b5b1dd54-6a88-46eb-8f13-0e7a54c9490e nodeName:}" failed. No retries permitted until 2026-09-11 18:12:43.861498062 +0000 UTC m=+162.462859220 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls") pod "dns-default-7d7v2" (UID: "b5b1dd54-6a88-46eb-8f13-0e7a54c9490e") : secret "dns-default-metrics-tls" not found
Sep 11 18:11:41.573197 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:41.573163    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7dcfb0b-9065-4c7e-a163-73844899ac4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:41.573197 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:41.573205    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7dcfb0b-9065-4c7e-a163-73844899ac4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:41.573757 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:41.573287    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:11:41.573757 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:41.573300    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:11:41.573757 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:41.573358    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7dcfb0b-9065-4c7e-a163-73844899ac4c-kube-rbac-proxy-cm podName:f7dcfb0b-9065-4c7e-a163-73844899ac4c nodeName:}" failed. No retries permitted until 2026-09-11 18:11:49.573336351 +0000 UTC m=+108.174697498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7dcfb0b-9065-4c7e-a163-73844899ac4c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9zc7s" (UID: "f7dcfb0b-9065-4c7e-a163-73844899ac4c") : configmap "kube-rbac-proxy" not found
Sep 11 18:11:41.573757 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:11:41.573378    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7dcfb0b-9065-4c7e-a163-73844899ac4c-insights-runtime-extractor-tls podName:f7dcfb0b-9065-4c7e-a163-73844899ac4c nodeName:}" failed. No retries permitted until 2026-09-11 18:11:49.573367904 +0000 UTC m=+108.174729049 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7dcfb0b-9065-4c7e-a163-73844899ac4c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9zc7s" (UID: "f7dcfb0b-9065-4c7e-a163-73844899ac4c") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:11:45.303317 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:45.303288    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qhvvw"
Sep 11 18:11:49.626742 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:49.626694    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7dcfb0b-9065-4c7e-a163-73844899ac4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:49.626742 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:49.626739    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7dcfb0b-9065-4c7e-a163-73844899ac4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:49.627225 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:49.627185    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7dcfb0b-9065-4c7e-a163-73844899ac4c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:49.628932 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:49.628913    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7dcfb0b-9065-4c7e-a163-73844899ac4c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9zc7s\" (UID: \"f7dcfb0b-9065-4c7e-a163-73844899ac4c\") " pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:49.753554 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:49.753517    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9zc7s"
Sep 11 18:11:50.029199 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:50.029166    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9zc7s"]
Sep 11 18:11:50.032285 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:11:50.032260    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7dcfb0b_9065_4c7e_a163_73844899ac4c.slice/crio-237ae3fa4ed4d1deecf883ae554c538d688ab3702ababd120644043aac31f40b WatchSource:0}: Error finding container 237ae3fa4ed4d1deecf883ae554c538d688ab3702ababd120644043aac31f40b: Status 404 returned error can't find the container with id 237ae3fa4ed4d1deecf883ae554c538d688ab3702ababd120644043aac31f40b
Sep 11 18:11:50.405677 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:50.405552    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9zc7s" event={"ID":"f7dcfb0b-9065-4c7e-a163-73844899ac4c","Type":"ContainerStarted","Data":"f7932faeb599eb222bde63d0faa2dc2ec9203dabd83909a7e08aa275e5990231"}
Sep 11 18:11:50.405831 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:50.405683    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9zc7s" event={"ID":"f7dcfb0b-9065-4c7e-a163-73844899ac4c","Type":"ContainerStarted","Data":"237ae3fa4ed4d1deecf883ae554c538d688ab3702ababd120644043aac31f40b"}
Sep 11 18:11:51.410230 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:51.410199    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9zc7s" event={"ID":"f7dcfb0b-9065-4c7e-a163-73844899ac4c","Type":"ContainerStarted","Data":"fa2e8c15e6f74cb41aa9f5dc7006cd2171b5ee29c6872ae214f7da9744597c84"}
Sep 11 18:11:53.417827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:53.417795    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9zc7s" event={"ID":"f7dcfb0b-9065-4c7e-a163-73844899ac4c","Type":"ContainerStarted","Data":"47ce42811985a7ac2daccc4eb30f21bc7a0d34273d731773f921f5b7e593168e"}
Sep 11 18:11:53.438101 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:11:53.438052    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9zc7s" podStartSLOduration=18.180928779 podStartE2EDuration="20.438039175s" totalImagesPullingTime="2.257110396s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:11:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:11:53.437572039 +0000 UTC m=+112.038933203" watchObservedRunningTime="2026-09-11 18:11:53.438039175 +0000 UTC m=+112.039400378"
Sep 11 18:12:06.158630 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:06.158570    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5" podUID="1570f52c-8164-4826-9168-30980530ceeb" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 11 18:12:11.784985 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:11.784952    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs\") pod \"network-metrics-daemon-44nnd\" (UID: \"6609afb5-5000-42c3-814e-9f9ee86a32d4\") " pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:12:11.787348 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:11.787324    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6609afb5-5000-42c3-814e-9f9ee86a32d4-metrics-certs\") pod \"network-metrics-daemon-44nnd\" (UID: \"6609afb5-5000-42c3-814e-9f9ee86a32d4\") " pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:12:12.017764 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.017734    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bvnsl\""
Sep 11 18:12:12.025535 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.025519    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-44nnd"
Sep 11 18:12:12.298672 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.298640    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-f8cpd"]
Sep 11 18:12:12.300879 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.300863    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.304784 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.304760    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 18:12:12.305752 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.305728    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 18:12:12.305752 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.305746    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 18:12:12.305920 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.305744    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 18:12:12.305920 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.305835    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 18:12:12.305920 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.305848    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xkt9g\""
Sep 11 18:12:12.306077 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.305937    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 18:12:12.315116 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.315098    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-44nnd"]
Sep 11 18:12:12.319081 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:12:12.319059    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6609afb5_5000_42c3_814e_9f9ee86a32d4.slice/crio-3507fe7fa1ccc9bd042d72db20f69fc43f919e7dba04a9c21900ff27f51fed27 WatchSource:0}: Error finding container 3507fe7fa1ccc9bd042d72db20f69fc43f919e7dba04a9c21900ff27f51fed27: Status 404 returned error can't find the container with id 3507fe7fa1ccc9bd042d72db20f69fc43f919e7dba04a9c21900ff27f51fed27
Sep 11 18:12:12.389414 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.389387    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4417b564-d430-4942-85f8-211e3098188e-root\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.389558 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.389418    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4417b564-d430-4942-85f8-211e3098188e-node-exporter-wtmp\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.389558 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.389435    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbqkg\" (UniqueName: \"kubernetes.io/projected/4417b564-d430-4942-85f8-211e3098188e-kube-api-access-lbqkg\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.389558 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.389452    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4417b564-d430-4942-85f8-211e3098188e-sys\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.389558 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.389514    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4417b564-d430-4942-85f8-211e3098188e-node-exporter-accelerators-collector-config\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.389715 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.389566    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4417b564-d430-4942-85f8-211e3098188e-metrics-client-ca\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.389715 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.389601    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4417b564-d430-4942-85f8-211e3098188e-node-exporter-tls\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.389715 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.389639    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4417b564-d430-4942-85f8-211e3098188e-node-exporter-textfile\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.389715 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.389670    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4417b564-d430-4942-85f8-211e3098188e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.475996 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.475961    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-44nnd" event={"ID":"6609afb5-5000-42c3-814e-9f9ee86a32d4","Type":"ContainerStarted","Data":"3507fe7fa1ccc9bd042d72db20f69fc43f919e7dba04a9c21900ff27f51fed27"}
Sep 11 18:12:12.490844 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.490820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4417b564-d430-4942-85f8-211e3098188e-root\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.490959 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.490850    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4417b564-d430-4942-85f8-211e3098188e-node-exporter-wtmp\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.490959 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.490866    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbqkg\" (UniqueName: \"kubernetes.io/projected/4417b564-d430-4942-85f8-211e3098188e-kube-api-access-lbqkg\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.490959 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.490888    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4417b564-d430-4942-85f8-211e3098188e-sys\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.490959 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.490917    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4417b564-d430-4942-85f8-211e3098188e-node-exporter-accelerators-collector-config\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.490959 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.490933    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4417b564-d430-4942-85f8-211e3098188e-root\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.491159 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.490989    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4417b564-d430-4942-85f8-211e3098188e-sys\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.491159 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.491023    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4417b564-d430-4942-85f8-211e3098188e-node-exporter-wtmp\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.491159 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.491073    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4417b564-d430-4942-85f8-211e3098188e-metrics-client-ca\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.491159 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.491111    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4417b564-d430-4942-85f8-211e3098188e-node-exporter-tls\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.491159 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.491156    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4417b564-d430-4942-85f8-211e3098188e-node-exporter-textfile\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.491337 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.491204    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4417b564-d430-4942-85f8-211e3098188e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.491501 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.491481    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4417b564-d430-4942-85f8-211e3098188e-node-exporter-textfile\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.491555 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.491481    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4417b564-d430-4942-85f8-211e3098188e-node-exporter-accelerators-collector-config\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.491651 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.491635    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4417b564-d430-4942-85f8-211e3098188e-metrics-client-ca\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.493475 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.493452    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4417b564-d430-4942-85f8-211e3098188e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.493570 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.493548    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4417b564-d430-4942-85f8-211e3098188e-node-exporter-tls\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.500234 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.500210    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbqkg\" (UniqueName: \"kubernetes.io/projected/4417b564-d430-4942-85f8-211e3098188e-kube-api-access-lbqkg\") pod \"node-exporter-f8cpd\" (UID: \"4417b564-d430-4942-85f8-211e3098188e\") " pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.609365 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:12.609300    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f8cpd"
Sep 11 18:12:12.618096 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:12:12.618066    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4417b564_d430_4942_85f8_211e3098188e.slice/crio-e39c016ac0e100baea554b4f1f175cd0b86d9c1c2c2225be5c474e84c5f3c995 WatchSource:0}: Error finding container e39c016ac0e100baea554b4f1f175cd0b86d9c1c2c2225be5c474e84c5f3c995: Status 404 returned error can't find the container with id e39c016ac0e100baea554b4f1f175cd0b86d9c1c2c2225be5c474e84c5f3c995
Sep 11 18:12:13.481836 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:13.481794    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-44nnd" event={"ID":"6609afb5-5000-42c3-814e-9f9ee86a32d4","Type":"ContainerStarted","Data":"4eba38bb7fd5c432411bf046cd543dcc96b00d9cbc86ec6d4ad937f25c9cd7d6"}
Sep 11 18:12:13.482291 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:13.481854    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-44nnd" event={"ID":"6609afb5-5000-42c3-814e-9f9ee86a32d4","Type":"ContainerStarted","Data":"db58a00eecd45fd828395578a8176c716c578fa23e496f4f3529f2ee17a13557"}
Sep 11 18:12:13.484216 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:13.484106    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f8cpd" event={"ID":"4417b564-d430-4942-85f8-211e3098188e","Type":"ContainerStarted","Data":"e39c016ac0e100baea554b4f1f175cd0b86d9c1c2c2225be5c474e84c5f3c995"}
Sep 11 18:12:13.500544 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:13.500500    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-44nnd" podStartSLOduration=130.695987897 podStartE2EDuration="2m11.500479107s" totalImagesPullingTime="804.49121ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:12:13.499364035 +0000 UTC m=+132.100725199" watchObservedRunningTime="2026-09-11 18:12:13.500479107 +0000 UTC m=+132.101840272"
Sep 11 18:12:14.489419 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:14.489382    2576 generic.go:356] "Generic (PLEG): container finished" podID="4417b564-d430-4942-85f8-211e3098188e" containerID="b31e0a90f057359b3f15a3349496129803339389daebdef5e556a39188071f30" exitCode=0
Sep 11 18:12:14.489820 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:14.489465    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f8cpd" event={"ID":"4417b564-d430-4942-85f8-211e3098188e","Type":"ContainerDied","Data":"b31e0a90f057359b3f15a3349496129803339389daebdef5e556a39188071f30"}
Sep 11 18:12:15.495156 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:15.495121    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f8cpd" event={"ID":"4417b564-d430-4942-85f8-211e3098188e","Type":"ContainerStarted","Data":"e0b3f79ef1fef51e72b7c3b9ecfe8929d2419dd7b73813698523b5d406a05105"}
Sep 11 18:12:15.495534 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:15.495171    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f8cpd" event={"ID":"4417b564-d430-4942-85f8-211e3098188e","Type":"ContainerStarted","Data":"352bc313fa5c3f43dae58a871b47e99aeadfd18bf3f33d650bbb0ef65c19559d"}
Sep 11 18:12:15.517792 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:15.517748    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-f8cpd" podStartSLOduration=2.393874833 podStartE2EDuration="3.517735082s" totalImagesPullingTime="1.064092866s" totalInitContainerRuntime="59.767383ms" isStatefulPod=true podCreationTimestamp="2026-09-11 18:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:12:15.517498056 +0000 UTC m=+134.118859232" watchObservedRunningTime="2026-09-11 18:12:15.517735082 +0000 UTC m=+134.119096236"
Sep 11 18:12:16.158141 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:16.158101    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5" podUID="1570f52c-8164-4826-9168-30980530ceeb" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 11 18:12:22.512970 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:22.512929    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"]
Sep 11 18:12:22.513425 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:12:22.513181    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6" podUID="c9add6aa-d540-4a0b-a893-e2e535c266ad"
Sep 11 18:12:22.518624 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:22.518573    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:12:22.526247 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:22.526225    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:12:22.566245 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:22.566208    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9add6aa-d540-4a0b-a893-e2e535c266ad-installation-pull-secrets\") pod \"c9add6aa-d540-4a0b-a893-e2e535c266ad\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") "
Sep 11 18:12:22.566412 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:22.566258    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkbsq\" (UniqueName: \"kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-kube-api-access-hkbsq\") pod \"c9add6aa-d540-4a0b-a893-e2e535c266ad\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") "
Sep 11 18:12:22.566412 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:22.566278    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9add6aa-d540-4a0b-a893-e2e535c266ad-trusted-ca\") pod \"c9add6aa-d540-4a0b-a893-e2e535c266ad\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") "
Sep 11 18:12:22.566412 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:22.566314    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-certificates\") pod \"c9add6aa-d540-4a0b-a893-e2e535c266ad\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") "
Sep 11 18:12:22.566412 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:22.566332    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9add6aa-d540-4a0b-a893-e2e535c266ad-image-registry-private-configuration\") pod \"c9add6aa-d540-4a0b-a893-e2e535c266ad\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") "
Sep 11 18:12:22.566412 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:22.566348    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-bound-sa-token\") pod \"c9add6aa-d540-4a0b-a893-e2e535c266ad\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") "
Sep 11 18:12:22.566412 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:22.566376    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9add6aa-d540-4a0b-a893-e2e535c266ad-ca-trust-extracted\") pod \"c9add6aa-d540-4a0b-a893-e2e535c266ad\" (UID: \"c9add6aa-d540-4a0b-a893-e2e535c266ad\") "
Sep 11 18:12:22.567201 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:22.566785    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c9add6aa-d540-4a0b-a893-e2e535c266ad" (UID: "c9add6aa-d540-4a0b-a893-e2e535c266ad"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:12:22.567201 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:22.566796    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9add6aa-d540-4a0b-a893-e2e535c266ad-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c9add6aa-d540-4a0b-a893-e2e535c266ad" (UID: "c9add6aa-d540-4a0b-a893-e2e535c266ad"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:12:22.567201 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:22.567075    2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-certificates\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:12:22.567201 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:22.567096    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9add6aa-d540-4a0b-a893-e2e535c266ad-trusted-ca\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:12:22.567451 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:22.567376    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9add6aa-d540-4a0b-a893-e2e535c266ad-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c9add6aa-d540-4a0b-a893-e2e535c266ad" (UID: "c9add6aa-d540-4a0b-a893-e2e535c266ad"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:12:22.569007 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:22.568974    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-kube-api-access-hkbsq" (OuterVolumeSpecName: "kube-api-access-hkbsq") pod "c9add6aa-d540-4a0b-a893-e2e535c266ad" (UID: "c9add6aa-d540-4a0b-a893-e2e535c266ad"). InnerVolumeSpecName "kube-api-access-hkbsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:12:22.569113 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:22.569008    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c9add6aa-d540-4a0b-a893-e2e535c266ad" (UID: "c9add6aa-d540-4a0b-a893-e2e535c266ad"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:12:22.569113 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:22.568980    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9add6aa-d540-4a0b-a893-e2e535c266ad-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c9add6aa-d540-4a0b-a893-e2e535c266ad" (UID: "c9add6aa-d540-4a0b-a893-e2e535c266ad"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:12:22.569113 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:22.569070    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9add6aa-d540-4a0b-a893-e2e535c266ad-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c9add6aa-d540-4a0b-a893-e2e535c266ad" (UID: "c9add6aa-d540-4a0b-a893-e2e535c266ad"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:12:22.668471 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:22.668431    2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9add6aa-d540-4a0b-a893-e2e535c266ad-installation-pull-secrets\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:12:22.668471 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:22.668462    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkbsq\" (UniqueName: \"kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-kube-api-access-hkbsq\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:12:22.668471 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:22.668473    2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9add6aa-d540-4a0b-a893-e2e535c266ad-image-registry-private-configuration\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:12:22.668471 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:22.668482    2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-bound-sa-token\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:12:22.668768 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:22.668491    2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9add6aa-d540-4a0b-a893-e2e535c266ad-ca-trust-extracted\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:12:23.521968 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:23.521936    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"
Sep 11 18:12:23.562136 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:23.562104    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"]
Sep 11 18:12:23.565835 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:23.565808    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6c8b6c9dbc-jq7q6"]
Sep 11 18:12:23.574083 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:23.574064    2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9add6aa-d540-4a0b-a893-e2e535c266ad-registry-tls\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:12:23.909388 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:23.909309    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9add6aa-d540-4a0b-a893-e2e535c266ad" path="/var/lib/kubelet/pods/c9add6aa-d540-4a0b-a893-e2e535c266ad/volumes"
Sep 11 18:12:26.158651 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:26.158609    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5" podUID="1570f52c-8164-4826-9168-30980530ceeb" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 11 18:12:26.159055 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:26.158684    2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5"
Sep 11 18:12:26.159273 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:26.159240    2576 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"1bf0778574efac507af54491361b033010a457aa2982b3f8af8bc19675109097"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 11 18:12:26.159320 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:26.159301    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5" podUID="1570f52c-8164-4826-9168-30980530ceeb" containerName="service-proxy" containerID="cri-o://1bf0778574efac507af54491361b033010a457aa2982b3f8af8bc19675109097" gracePeriod=30
Sep 11 18:12:26.533760 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:26.533727    2576 generic.go:356] "Generic (PLEG): container finished" podID="1570f52c-8164-4826-9168-30980530ceeb" containerID="1bf0778574efac507af54491361b033010a457aa2982b3f8af8bc19675109097" exitCode=2
Sep 11 18:12:26.533959 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:26.533790    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5" event={"ID":"1570f52c-8164-4826-9168-30980530ceeb","Type":"ContainerDied","Data":"1bf0778574efac507af54491361b033010a457aa2982b3f8af8bc19675109097"}
Sep 11 18:12:26.533959 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:26.533822    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c86957f4c-99xh5" event={"ID":"1570f52c-8164-4826-9168-30980530ceeb","Type":"ContainerStarted","Data":"1634e36303e24ea7c2c9f10190e788f07a557353a0962e2d768a55cf6c898373"}
Sep 11 18:12:38.925278 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:12:38.925237    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7d7v2" podUID="b5b1dd54-6a88-46eb-8f13-0e7a54c9490e"
Sep 11 18:12:39.578665 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:39.578636    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:12:43.911309 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:43.911279    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls\") pod \"dns-default-7d7v2\" (UID: \"b5b1dd54-6a88-46eb-8f13-0e7a54c9490e\") " pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:12:43.913514 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:43.913487    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5b1dd54-6a88-46eb-8f13-0e7a54c9490e-metrics-tls\") pod \"dns-default-7d7v2\" (UID: \"b5b1dd54-6a88-46eb-8f13-0e7a54c9490e\") " pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:12:44.081405 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:44.081372    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4zn8w\""
Sep 11 18:12:44.089952 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:44.089933    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:12:44.369816 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:44.369787    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7d7v2"]
Sep 11 18:12:44.372771 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:12:44.372737    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b1dd54_6a88_46eb_8f13_0e7a54c9490e.slice/crio-281eeef869faa8a3ebe077f051448cc163f829a4c73ad70c8b90ed6ac2ba8739 WatchSource:0}: Error finding container 281eeef869faa8a3ebe077f051448cc163f829a4c73ad70c8b90ed6ac2ba8739: Status 404 returned error can't find the container with id 281eeef869faa8a3ebe077f051448cc163f829a4c73ad70c8b90ed6ac2ba8739
Sep 11 18:12:44.594614 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:44.594564    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7d7v2" event={"ID":"b5b1dd54-6a88-46eb-8f13-0e7a54c9490e","Type":"ContainerStarted","Data":"281eeef869faa8a3ebe077f051448cc163f829a4c73ad70c8b90ed6ac2ba8739"}
Sep 11 18:12:46.603467 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:46.603432    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7d7v2" event={"ID":"b5b1dd54-6a88-46eb-8f13-0e7a54c9490e","Type":"ContainerStarted","Data":"9b4c3f73c798eafb98bf51446a78e400ee3b25290c97f3283ffd8ba2657d0265"}
Sep 11 18:12:46.603849 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:46.603488    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7d7v2" event={"ID":"b5b1dd54-6a88-46eb-8f13-0e7a54c9490e","Type":"ContainerStarted","Data":"e51aea5394d2ae870b317f7e363193cbde1c1247e9a9adf2ee5e4db00c7582a6"}
Sep 11 18:12:46.603849 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:46.603531    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:12:56.609327 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:56.609238    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7d7v2"
Sep 11 18:12:56.635162 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:12:56.635109    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7d7v2" podStartSLOduration=140.104387377 podStartE2EDuration="2m21.635094386s" totalImagesPullingTime="1.530707009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:12:46.638808042 +0000 UTC m=+165.240169206" watchObservedRunningTime="2026-09-11 18:12:56.635094386 +0000 UTC m=+175.236455548"
Sep 11 18:15:01.827007 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:01.826975    2576 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 18:15:20.651486 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:20.651447    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9wzn5"]
Sep 11 18:15:20.654776 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:20.654757    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-9wzn5"
Sep 11 18:15:20.658161 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:20.658142    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 18:15:20.658254 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:20.658238    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-czrcc\""
Sep 11 18:15:20.659286 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:20.659250    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 18:15:20.670808 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:20.670789    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9wzn5"]
Sep 11 18:15:20.769719 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:20.769684    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qp9v\" (UniqueName: \"kubernetes.io/projected/576727a8-a399-4220-a67b-93f309bd4223-kube-api-access-2qp9v\") pod \"cert-manager-55b4d954d5-9wzn5\" (UID: \"576727a8-a399-4220-a67b-93f309bd4223\") " pod="cert-manager/cert-manager-55b4d954d5-9wzn5"
Sep 11 18:15:20.769891 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:20.769733    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/576727a8-a399-4220-a67b-93f309bd4223-bound-sa-token\") pod \"cert-manager-55b4d954d5-9wzn5\" (UID: \"576727a8-a399-4220-a67b-93f309bd4223\") " pod="cert-manager/cert-manager-55b4d954d5-9wzn5"
Sep 11 18:15:20.870121 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:20.870082    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qp9v\" (UniqueName: \"kubernetes.io/projected/576727a8-a399-4220-a67b-93f309bd4223-kube-api-access-2qp9v\") pod \"cert-manager-55b4d954d5-9wzn5\" (UID: \"576727a8-a399-4220-a67b-93f309bd4223\") " pod="cert-manager/cert-manager-55b4d954d5-9wzn5"
Sep 11 18:15:20.870300 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:20.870135    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/576727a8-a399-4220-a67b-93f309bd4223-bound-sa-token\") pod \"cert-manager-55b4d954d5-9wzn5\" (UID: \"576727a8-a399-4220-a67b-93f309bd4223\") " pod="cert-manager/cert-manager-55b4d954d5-9wzn5"
Sep 11 18:15:20.879723 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:20.879702    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/576727a8-a399-4220-a67b-93f309bd4223-bound-sa-token\") pod \"cert-manager-55b4d954d5-9wzn5\" (UID: \"576727a8-a399-4220-a67b-93f309bd4223\") " pod="cert-manager/cert-manager-55b4d954d5-9wzn5"
Sep 11 18:15:20.879845 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:20.879778    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qp9v\" (UniqueName: \"kubernetes.io/projected/576727a8-a399-4220-a67b-93f309bd4223-kube-api-access-2qp9v\") pod \"cert-manager-55b4d954d5-9wzn5\" (UID: \"576727a8-a399-4220-a67b-93f309bd4223\") " pod="cert-manager/cert-manager-55b4d954d5-9wzn5"
Sep 11 18:15:20.962812 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:20.962783    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-9wzn5"
Sep 11 18:15:21.233285 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:21.233242    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9wzn5"]
Sep 11 18:15:21.235415 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:15:21.235390    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod576727a8_a399_4220_a67b_93f309bd4223.slice/crio-1ddb6687fafc9449153da2f5ef3299efb536131016f4e7140df68828e16329c8 WatchSource:0}: Error finding container 1ddb6687fafc9449153da2f5ef3299efb536131016f4e7140df68828e16329c8: Status 404 returned error can't find the container with id 1ddb6687fafc9449153da2f5ef3299efb536131016f4e7140df68828e16329c8
Sep 11 18:15:21.237325 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:21.237304    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:15:21.679783 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:21.679704    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fdcmk"]
Sep 11 18:15:21.682894 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:21.682869    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fdcmk"
Sep 11 18:15:21.686153 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:21.686130    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-xkcns\""
Sep 11 18:15:21.686278 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:21.686128    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 18:15:21.686832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:21.686814    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:15:21.696672 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:21.696648    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fdcmk"]
Sep 11 18:15:21.778191 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:21.778156    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2xtf\" (UniqueName: \"kubernetes.io/projected/14ba344f-31dc-4a15-882e-9ef608ece7d4-kube-api-access-n2xtf\") pod \"openshift-lws-operator-fd8ccff4c-fdcmk\" (UID: \"14ba344f-31dc-4a15-882e-9ef608ece7d4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fdcmk"
Sep 11 18:15:21.778357 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:21.778202    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14ba344f-31dc-4a15-882e-9ef608ece7d4-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fdcmk\" (UID: \"14ba344f-31dc-4a15-882e-9ef608ece7d4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fdcmk"
Sep 11 18:15:21.879244 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:21.879205    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2xtf\" (UniqueName: \"kubernetes.io/projected/14ba344f-31dc-4a15-882e-9ef608ece7d4-kube-api-access-n2xtf\") pod \"openshift-lws-operator-fd8ccff4c-fdcmk\" (UID: \"14ba344f-31dc-4a15-882e-9ef608ece7d4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fdcmk"
Sep 11 18:15:21.879427 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:21.879272    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14ba344f-31dc-4a15-882e-9ef608ece7d4-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fdcmk\" (UID: \"14ba344f-31dc-4a15-882e-9ef608ece7d4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fdcmk"
Sep 11 18:15:21.879776 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:21.879731    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14ba344f-31dc-4a15-882e-9ef608ece7d4-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fdcmk\" (UID: \"14ba344f-31dc-4a15-882e-9ef608ece7d4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fdcmk"
Sep 11 18:15:21.893133 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:21.893086    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2xtf\" (UniqueName: \"kubernetes.io/projected/14ba344f-31dc-4a15-882e-9ef608ece7d4-kube-api-access-n2xtf\") pod \"openshift-lws-operator-fd8ccff4c-fdcmk\" (UID: \"14ba344f-31dc-4a15-882e-9ef608ece7d4\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fdcmk"
Sep 11 18:15:21.996050 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:21.995979    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fdcmk"
Sep 11 18:15:22.101994 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:22.101953    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-9wzn5" event={"ID":"576727a8-a399-4220-a67b-93f309bd4223","Type":"ContainerStarted","Data":"1ddb6687fafc9449153da2f5ef3299efb536131016f4e7140df68828e16329c8"}
Sep 11 18:15:22.291439 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:22.291411    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fdcmk"]
Sep 11 18:15:22.294559 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:15:22.294529    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14ba344f_31dc_4a15_882e_9ef608ece7d4.slice/crio-6f34764a3137978135ab05b779918240f709935bc1d71fed1770ec8edaa2b256 WatchSource:0}: Error finding container 6f34764a3137978135ab05b779918240f709935bc1d71fed1770ec8edaa2b256: Status 404 returned error can't find the container with id 6f34764a3137978135ab05b779918240f709935bc1d71fed1770ec8edaa2b256
Sep 11 18:15:23.106676 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:23.106639    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fdcmk" event={"ID":"14ba344f-31dc-4a15-882e-9ef608ece7d4","Type":"ContainerStarted","Data":"6f34764a3137978135ab05b779918240f709935bc1d71fed1770ec8edaa2b256"}
Sep 11 18:15:25.117228 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:25.117191    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-9wzn5" event={"ID":"576727a8-a399-4220-a67b-93f309bd4223","Type":"ContainerStarted","Data":"aa2e53570d18a659d50ae76c62132b291cdfd34732167104de0fe3988804586c"}
Sep 11 18:15:25.138197 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:25.138145    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-9wzn5" podStartSLOduration=2.321547001 podStartE2EDuration="5.138123729s" totalImagesPullingTime="2.816576728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:25.137228159 +0000 UTC m=+323.738589323" watchObservedRunningTime="2026-09-11 18:15:25.138123729 +0000 UTC m=+323.739484893"
Sep 11 18:15:26.122570 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:26.122529    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fdcmk" event={"ID":"14ba344f-31dc-4a15-882e-9ef608ece7d4","Type":"ContainerStarted","Data":"bc6a3e2eaefcc2e06a0f65670e8c382a4c9c3e0b74fbeaa97b14d588f2437445"}
Sep 11 18:15:26.144045 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:26.144000    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fdcmk" podStartSLOduration=1.504742767 podStartE2EDuration="5.143985385s" totalImagesPullingTime="3.639242618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:26.142410545 +0000 UTC m=+324.743771749" watchObservedRunningTime="2026-09-11 18:15:26.143985385 +0000 UTC m=+324.745346548"
Sep 11 18:15:41.880655 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:41.880616    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6"]
Sep 11 18:15:41.886115 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:41.886090    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6"
Sep 11 18:15:41.888531 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:41.888500    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 18:15:41.888736 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:41.888724    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 18:15:41.888798 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:41.888787    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 18:15:41.889229 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:41.889213    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-m5jv4\""
Sep 11 18:15:41.889273 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:41.889216    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 18:15:41.909017 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:41.908988    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6"]
Sep 11 18:15:42.029031 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:42.028993    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efc58b08-02fd-4836-a9b0-9dab7ce86dec-tmp\") pod \"opendatahub-operator-controller-manager-657fcdcb9c-5nvr6\" (UID: \"efc58b08-02fd-4836-a9b0-9dab7ce86dec\") " pod="opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6"
Sep 11 18:15:42.029031 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:42.029028    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/efc58b08-02fd-4836-a9b0-9dab7ce86dec-webhook-cert\") pod \"opendatahub-operator-controller-manager-657fcdcb9c-5nvr6\" (UID: \"efc58b08-02fd-4836-a9b0-9dab7ce86dec\") " pod="opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6"
Sep 11 18:15:42.029299 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:42.029048    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/efc58b08-02fd-4836-a9b0-9dab7ce86dec-manifests\") pod \"opendatahub-operator-controller-manager-657fcdcb9c-5nvr6\" (UID: \"efc58b08-02fd-4836-a9b0-9dab7ce86dec\") " pod="opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6"
Sep 11 18:15:42.029299 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:42.029064    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/efc58b08-02fd-4836-a9b0-9dab7ce86dec-apiservice-cert\") pod \"opendatahub-operator-controller-manager-657fcdcb9c-5nvr6\" (UID: \"efc58b08-02fd-4836-a9b0-9dab7ce86dec\") " pod="opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6"
Sep 11 18:15:42.029299 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:42.029208    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nndm8\" (UniqueName: \"kubernetes.io/projected/efc58b08-02fd-4836-a9b0-9dab7ce86dec-kube-api-access-nndm8\") pod \"opendatahub-operator-controller-manager-657fcdcb9c-5nvr6\" (UID: \"efc58b08-02fd-4836-a9b0-9dab7ce86dec\") " pod="opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6"
Sep 11 18:15:42.129755 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:42.129716    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nndm8\" (UniqueName: \"kubernetes.io/projected/efc58b08-02fd-4836-a9b0-9dab7ce86dec-kube-api-access-nndm8\") pod \"opendatahub-operator-controller-manager-657fcdcb9c-5nvr6\" (UID: \"efc58b08-02fd-4836-a9b0-9dab7ce86dec\") " pod="opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6"
Sep 11 18:15:42.129940 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:42.129774    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efc58b08-02fd-4836-a9b0-9dab7ce86dec-tmp\") pod \"opendatahub-operator-controller-manager-657fcdcb9c-5nvr6\" (UID: \"efc58b08-02fd-4836-a9b0-9dab7ce86dec\") " pod="opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6"
Sep 11 18:15:42.130004 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:42.129941    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/efc58b08-02fd-4836-a9b0-9dab7ce86dec-webhook-cert\") pod \"opendatahub-operator-controller-manager-657fcdcb9c-5nvr6\" (UID: \"efc58b08-02fd-4836-a9b0-9dab7ce86dec\") " pod="opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6"
Sep 11 18:15:42.130004 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:42.129972    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/efc58b08-02fd-4836-a9b0-9dab7ce86dec-manifests\") pod \"opendatahub-operator-controller-manager-657fcdcb9c-5nvr6\" (UID: \"efc58b08-02fd-4836-a9b0-9dab7ce86dec\") " pod="opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6"
Sep 11 18:15:42.130004 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:42.129989    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/efc58b08-02fd-4836-a9b0-9dab7ce86dec-apiservice-cert\") pod \"opendatahub-operator-controller-manager-657fcdcb9c-5nvr6\" (UID: \"efc58b08-02fd-4836-a9b0-9dab7ce86dec\") " pod="opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6"
Sep 11 18:15:42.130329 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:42.130304    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/efc58b08-02fd-4836-a9b0-9dab7ce86dec-manifests\") pod \"opendatahub-operator-controller-manager-657fcdcb9c-5nvr6\" (UID: \"efc58b08-02fd-4836-a9b0-9dab7ce86dec\") " pod="opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6"
Sep 11 18:15:42.132266 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:42.132201    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efc58b08-02fd-4836-a9b0-9dab7ce86dec-tmp\") pod \"opendatahub-operator-controller-manager-657fcdcb9c-5nvr6\" (UID: \"efc58b08-02fd-4836-a9b0-9dab7ce86dec\") " pod="opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6"
Sep 11 18:15:42.132266 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:42.132249    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/efc58b08-02fd-4836-a9b0-9dab7ce86dec-apiservice-cert\") pod \"opendatahub-operator-controller-manager-657fcdcb9c-5nvr6\" (UID: \"efc58b08-02fd-4836-a9b0-9dab7ce86dec\") " pod="opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6"
Sep 11 18:15:42.132425 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:42.132359    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/efc58b08-02fd-4836-a9b0-9dab7ce86dec-webhook-cert\") pod \"opendatahub-operator-controller-manager-657fcdcb9c-5nvr6\" (UID: \"efc58b08-02fd-4836-a9b0-9dab7ce86dec\") " pod="opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6"
Sep 11 18:15:42.138676 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:42.138653    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nndm8\" (UniqueName: \"kubernetes.io/projected/efc58b08-02fd-4836-a9b0-9dab7ce86dec-kube-api-access-nndm8\") pod \"opendatahub-operator-controller-manager-657fcdcb9c-5nvr6\" (UID: \"efc58b08-02fd-4836-a9b0-9dab7ce86dec\") " pod="opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6"
Sep 11 18:15:42.195747 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:42.195704    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6"
Sep 11 18:15:42.479258 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:42.479223    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6"]
Sep 11 18:15:42.482749 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:15:42.482710    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc58b08_02fd_4836_a9b0_9dab7ce86dec.slice/crio-71e37aae169557dceaba5f4f55e3ec9b2b43405d87c0ee7da2a4eb7dc58be316 WatchSource:0}: Error finding container 71e37aae169557dceaba5f4f55e3ec9b2b43405d87c0ee7da2a4eb7dc58be316: Status 404 returned error can't find the container with id 71e37aae169557dceaba5f4f55e3ec9b2b43405d87c0ee7da2a4eb7dc58be316
Sep 11 18:15:43.185331 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:43.185295    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6" event={"ID":"efc58b08-02fd-4836-a9b0-9dab7ce86dec","Type":"ContainerStarted","Data":"71e37aae169557dceaba5f4f55e3ec9b2b43405d87c0ee7da2a4eb7dc58be316"}
Sep 11 18:15:46.198339 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:46.198300    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6" event={"ID":"efc58b08-02fd-4836-a9b0-9dab7ce86dec","Type":"ContainerStarted","Data":"d401b6553c9a5b20ab5b35b2b764e0bffb9f5cfedb12d3e7c449cac8cd031b7a"}
Sep 11 18:15:47.203233 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:47.203200    2576 generic.go:356] "Generic (PLEG): container finished" podID="efc58b08-02fd-4836-a9b0-9dab7ce86dec" containerID="d401b6553c9a5b20ab5b35b2b764e0bffb9f5cfedb12d3e7c449cac8cd031b7a" exitCode=0
Sep 11 18:15:47.203706 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:47.203278    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6" event={"ID":"efc58b08-02fd-4836-a9b0-9dab7ce86dec","Type":"ContainerDied","Data":"d401b6553c9a5b20ab5b35b2b764e0bffb9f5cfedb12d3e7c449cac8cd031b7a"}
Sep 11 18:15:48.208410 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:48.208376    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6" event={"ID":"efc58b08-02fd-4836-a9b0-9dab7ce86dec","Type":"ContainerStarted","Data":"8a753eb2eb5ba1b073e0a6fcb3aeb30c870fe86de0da250e9aa9d2ecfefffe45"}
Sep 11 18:15:48.208798 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:48.208449    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6"
Sep 11 18:15:48.243072 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:48.243019    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6" podStartSLOduration=2.430062385 podStartE2EDuration="7.243000376s" totalImagesPullingTime="3.787362543s" totalInitContainerRuntime="1.025575448s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:48.242099767 +0000 UTC m=+346.843460940" watchObservedRunningTime="2026-09-11 18:15:48.243000376 +0000 UTC m=+346.844361538"
Sep 11 18:15:59.214970 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:15:59.214887    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-657fcdcb9c-5nvr6"
Sep 11 18:16:02.643547 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:02.643511    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6dd5f9dbb5-qb7nd"]
Sep 11 18:16:02.647103 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:02.647086    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6dd5f9dbb5-qb7nd"
Sep 11 18:16:02.649979 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:02.649952    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 18:16:02.650101 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:02.649953    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 18:16:02.651373 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:02.651350    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-7c54n\""
Sep 11 18:16:02.651491 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:02.651429    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 18:16:02.669194 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:02.669160    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6dd5f9dbb5-qb7nd"]
Sep 11 18:16:02.789717 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:02.789673    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e0e19b41-5055-4885-9d70-df53bb3a9e87-manager-config\") pod \"lws-controller-manager-6dd5f9dbb5-qb7nd\" (UID: \"e0e19b41-5055-4885-9d70-df53bb3a9e87\") " pod="openshift-lws-operator/lws-controller-manager-6dd5f9dbb5-qb7nd"
Sep 11 18:16:02.789717 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:02.789721    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0e19b41-5055-4885-9d70-df53bb3a9e87-metrics-cert\") pod \"lws-controller-manager-6dd5f9dbb5-qb7nd\" (UID: \"e0e19b41-5055-4885-9d70-df53bb3a9e87\") " pod="openshift-lws-operator/lws-controller-manager-6dd5f9dbb5-qb7nd"
Sep 11 18:16:02.789969 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:02.789753    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jj2r\" (UniqueName: \"kubernetes.io/projected/e0e19b41-5055-4885-9d70-df53bb3a9e87-kube-api-access-6jj2r\") pod \"lws-controller-manager-6dd5f9dbb5-qb7nd\" (UID: \"e0e19b41-5055-4885-9d70-df53bb3a9e87\") " pod="openshift-lws-operator/lws-controller-manager-6dd5f9dbb5-qb7nd"
Sep 11 18:16:02.789969 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:02.789841    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0e19b41-5055-4885-9d70-df53bb3a9e87-cert\") pod \"lws-controller-manager-6dd5f9dbb5-qb7nd\" (UID: \"e0e19b41-5055-4885-9d70-df53bb3a9e87\") " pod="openshift-lws-operator/lws-controller-manager-6dd5f9dbb5-qb7nd"
Sep 11 18:16:02.890312 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:02.890274    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0e19b41-5055-4885-9d70-df53bb3a9e87-cert\") pod \"lws-controller-manager-6dd5f9dbb5-qb7nd\" (UID: \"e0e19b41-5055-4885-9d70-df53bb3a9e87\") " pod="openshift-lws-operator/lws-controller-manager-6dd5f9dbb5-qb7nd"
Sep 11 18:16:02.890502 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:02.890323    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e0e19b41-5055-4885-9d70-df53bb3a9e87-manager-config\") pod \"lws-controller-manager-6dd5f9dbb5-qb7nd\" (UID: \"e0e19b41-5055-4885-9d70-df53bb3a9e87\") " pod="openshift-lws-operator/lws-controller-manager-6dd5f9dbb5-qb7nd"
Sep 11 18:16:02.890502 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:02.890347    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0e19b41-5055-4885-9d70-df53bb3a9e87-metrics-cert\") pod \"lws-controller-manager-6dd5f9dbb5-qb7nd\" (UID: \"e0e19b41-5055-4885-9d70-df53bb3a9e87\") " pod="openshift-lws-operator/lws-controller-manager-6dd5f9dbb5-qb7nd"
Sep 11 18:16:02.890502 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:02.890412    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jj2r\" (UniqueName: \"kubernetes.io/projected/e0e19b41-5055-4885-9d70-df53bb3a9e87-kube-api-access-6jj2r\") pod \"lws-controller-manager-6dd5f9dbb5-qb7nd\" (UID: \"e0e19b41-5055-4885-9d70-df53bb3a9e87\") " pod="openshift-lws-operator/lws-controller-manager-6dd5f9dbb5-qb7nd"
Sep 11 18:16:02.890954 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:02.890935    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e0e19b41-5055-4885-9d70-df53bb3a9e87-manager-config\") pod \"lws-controller-manager-6dd5f9dbb5-qb7nd\" (UID: \"e0e19b41-5055-4885-9d70-df53bb3a9e87\") " pod="openshift-lws-operator/lws-controller-manager-6dd5f9dbb5-qb7nd"
Sep 11 18:16:02.892914 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:02.892894    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0e19b41-5055-4885-9d70-df53bb3a9e87-metrics-cert\") pod \"lws-controller-manager-6dd5f9dbb5-qb7nd\" (UID: \"e0e19b41-5055-4885-9d70-df53bb3a9e87\") " pod="openshift-lws-operator/lws-controller-manager-6dd5f9dbb5-qb7nd"
Sep 11 18:16:02.893016 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:02.892943    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0e19b41-5055-4885-9d70-df53bb3a9e87-cert\") pod \"lws-controller-manager-6dd5f9dbb5-qb7nd\" (UID: \"e0e19b41-5055-4885-9d70-df53bb3a9e87\") " pod="openshift-lws-operator/lws-controller-manager-6dd5f9dbb5-qb7nd"
Sep 11 18:16:02.910340 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:02.910271    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jj2r\" (UniqueName: \"kubernetes.io/projected/e0e19b41-5055-4885-9d70-df53bb3a9e87-kube-api-access-6jj2r\") pod \"lws-controller-manager-6dd5f9dbb5-qb7nd\" (UID: \"e0e19b41-5055-4885-9d70-df53bb3a9e87\") " pod="openshift-lws-operator/lws-controller-manager-6dd5f9dbb5-qb7nd"
Sep 11 18:16:02.956337 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:02.956288    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6dd5f9dbb5-qb7nd"
Sep 11 18:16:03.251442 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:03.251407    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6dd5f9dbb5-qb7nd"]
Sep 11 18:16:03.254554 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:16:03.254529    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e19b41_5055_4885_9d70_df53bb3a9e87.slice/crio-861cf9e0ec73aa2dce89a50e81bf95dec968fbe9dcdff2d6f3f2e0e070bc58e0 WatchSource:0}: Error finding container 861cf9e0ec73aa2dce89a50e81bf95dec968fbe9dcdff2d6f3f2e0e070bc58e0: Status 404 returned error can't find the container with id 861cf9e0ec73aa2dce89a50e81bf95dec968fbe9dcdff2d6f3f2e0e070bc58e0
Sep 11 18:16:03.259523 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:03.259496    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6dd5f9dbb5-qb7nd" event={"ID":"e0e19b41-5055-4885-9d70-df53bb3a9e87","Type":"ContainerStarted","Data":"861cf9e0ec73aa2dce89a50e81bf95dec968fbe9dcdff2d6f3f2e0e070bc58e0"}
Sep 11 18:16:05.268339 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:05.268302    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6dd5f9dbb5-qb7nd" event={"ID":"e0e19b41-5055-4885-9d70-df53bb3a9e87","Type":"ContainerStarted","Data":"6327fe908448a1678e8cbad2746f60e9a00afd4800d378816a2af09439e0daad"}
Sep 11 18:16:05.268732 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:05.268433    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6dd5f9dbb5-qb7nd"
Sep 11 18:16:05.293655 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:05.293599    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6dd5f9dbb5-qb7nd" podStartSLOduration=1.7701905249999998 podStartE2EDuration="3.293567946s" totalImagesPullingTime="1.523377421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:05.29190651 +0000 UTC m=+363.893267668" watchObservedRunningTime="2026-09-11 18:16:05.293567946 +0000 UTC m=+363.894929108"
Sep 11 18:16:12.710187 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:12.710139    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rdq5p"]
Sep 11 18:16:12.713307 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:12.713289    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rdq5p"
Sep 11 18:16:12.717208 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:12.717178    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 11 18:16:12.717722 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:12.717690    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-lzhh7\""
Sep 11 18:16:12.729033 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:12.728997    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rdq5p"]
Sep 11 18:16:12.769419 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:12.769379    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f99264f-7197-4667-b258-a27550fb9ad1-cert\") pod \"odh-model-controller-7998c67dc4-rdq5p\" (UID: \"7f99264f-7197-4667-b258-a27550fb9ad1\") " pod="opendatahub/odh-model-controller-7998c67dc4-rdq5p"
Sep 11 18:16:12.769646 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:12.769495    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkxs6\" (UniqueName: \"kubernetes.io/projected/7f99264f-7197-4667-b258-a27550fb9ad1-kube-api-access-pkxs6\") pod \"odh-model-controller-7998c67dc4-rdq5p\" (UID: \"7f99264f-7197-4667-b258-a27550fb9ad1\") " pod="opendatahub/odh-model-controller-7998c67dc4-rdq5p"
Sep 11 18:16:12.870459 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:12.870417    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkxs6\" (UniqueName: \"kubernetes.io/projected/7f99264f-7197-4667-b258-a27550fb9ad1-kube-api-access-pkxs6\") pod \"odh-model-controller-7998c67dc4-rdq5p\" (UID: \"7f99264f-7197-4667-b258-a27550fb9ad1\") " pod="opendatahub/odh-model-controller-7998c67dc4-rdq5p"
Sep 11 18:16:12.870459 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:12.870463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f99264f-7197-4667-b258-a27550fb9ad1-cert\") pod \"odh-model-controller-7998c67dc4-rdq5p\" (UID: \"7f99264f-7197-4667-b258-a27550fb9ad1\") " pod="opendatahub/odh-model-controller-7998c67dc4-rdq5p"
Sep 11 18:16:12.870760 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:16:12.870577    2576 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 11 18:16:12.870760 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:16:12.870672    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f99264f-7197-4667-b258-a27550fb9ad1-cert podName:7f99264f-7197-4667-b258-a27550fb9ad1 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:13.370655554 +0000 UTC m=+371.972016709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7f99264f-7197-4667-b258-a27550fb9ad1-cert") pod "odh-model-controller-7998c67dc4-rdq5p" (UID: "7f99264f-7197-4667-b258-a27550fb9ad1") : secret "odh-model-controller-webhook-cert" not found
Sep 11 18:16:12.889873 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:12.889844    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkxs6\" (UniqueName: \"kubernetes.io/projected/7f99264f-7197-4667-b258-a27550fb9ad1-kube-api-access-pkxs6\") pod \"odh-model-controller-7998c67dc4-rdq5p\" (UID: \"7f99264f-7197-4667-b258-a27550fb9ad1\") " pod="opendatahub/odh-model-controller-7998c67dc4-rdq5p"
Sep 11 18:16:13.352321 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:13.352282    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-qvlvs"]
Sep 11 18:16:13.355260 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:13.355238    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-qvlvs"
Sep 11 18:16:13.358174 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:13.358151    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 11 18:16:13.366298 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:13.366271    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-qvlvs"]
Sep 11 18:16:13.374500 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:13.374472    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2f4l\" (UniqueName: \"kubernetes.io/projected/f131341e-13f0-46a6-bf5a-54c35f749073-kube-api-access-s2f4l\") pod \"model-serving-api-54d89bd5fc-qvlvs\" (UID: \"f131341e-13f0-46a6-bf5a-54c35f749073\") " pod="opendatahub/model-serving-api-54d89bd5fc-qvlvs"
Sep 11 18:16:13.374660 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:13.374513    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f131341e-13f0-46a6-bf5a-54c35f749073-tls-certs\") pod \"model-serving-api-54d89bd5fc-qvlvs\" (UID: \"f131341e-13f0-46a6-bf5a-54c35f749073\") " pod="opendatahub/model-serving-api-54d89bd5fc-qvlvs"
Sep 11 18:16:13.374660 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:13.374548    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f99264f-7197-4667-b258-a27550fb9ad1-cert\") pod \"odh-model-controller-7998c67dc4-rdq5p\" (UID: \"7f99264f-7197-4667-b258-a27550fb9ad1\") " pod="opendatahub/odh-model-controller-7998c67dc4-rdq5p"
Sep 11 18:16:13.377128 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:13.377095    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f99264f-7197-4667-b258-a27550fb9ad1-cert\") pod \"odh-model-controller-7998c67dc4-rdq5p\" (UID: \"7f99264f-7197-4667-b258-a27550fb9ad1\") " pod="opendatahub/odh-model-controller-7998c67dc4-rdq5p"
Sep 11 18:16:13.475533 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:13.475483    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2f4l\" (UniqueName: \"kubernetes.io/projected/f131341e-13f0-46a6-bf5a-54c35f749073-kube-api-access-s2f4l\") pod \"model-serving-api-54d89bd5fc-qvlvs\" (UID: \"f131341e-13f0-46a6-bf5a-54c35f749073\") " pod="opendatahub/model-serving-api-54d89bd5fc-qvlvs"
Sep 11 18:16:13.475533 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:13.475535    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f131341e-13f0-46a6-bf5a-54c35f749073-tls-certs\") pod \"model-serving-api-54d89bd5fc-qvlvs\" (UID: \"f131341e-13f0-46a6-bf5a-54c35f749073\") " pod="opendatahub/model-serving-api-54d89bd5fc-qvlvs"
Sep 11 18:16:13.475793 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:16:13.475653    2576 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 11 18:16:13.475793 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:16:13.475716    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f131341e-13f0-46a6-bf5a-54c35f749073-tls-certs podName:f131341e-13f0-46a6-bf5a-54c35f749073 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:13.975699977 +0000 UTC m=+372.577061118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f131341e-13f0-46a6-bf5a-54c35f749073-tls-certs") pod "model-serving-api-54d89bd5fc-qvlvs" (UID: "f131341e-13f0-46a6-bf5a-54c35f749073") : secret "model-serving-api-tls" not found
Sep 11 18:16:13.486675 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:13.486645    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2f4l\" (UniqueName: \"kubernetes.io/projected/f131341e-13f0-46a6-bf5a-54c35f749073-kube-api-access-s2f4l\") pod \"model-serving-api-54d89bd5fc-qvlvs\" (UID: \"f131341e-13f0-46a6-bf5a-54c35f749073\") " pod="opendatahub/model-serving-api-54d89bd5fc-qvlvs"
Sep 11 18:16:13.623795 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:13.623698    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rdq5p"
Sep 11 18:16:13.909657 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:13.909559    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rdq5p"]
Sep 11 18:16:13.913470 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:16:13.913439    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f99264f_7197_4667_b258_a27550fb9ad1.slice/crio-fbf28d4353ae2dad2f9ae36bbb26e294e1a78b579dfb5b12cfb06a0d53abbd6d WatchSource:0}: Error finding container fbf28d4353ae2dad2f9ae36bbb26e294e1a78b579dfb5b12cfb06a0d53abbd6d: Status 404 returned error can't find the container with id fbf28d4353ae2dad2f9ae36bbb26e294e1a78b579dfb5b12cfb06a0d53abbd6d
Sep 11 18:16:13.980275 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:13.980222    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f131341e-13f0-46a6-bf5a-54c35f749073-tls-certs\") pod \"model-serving-api-54d89bd5fc-qvlvs\" (UID: \"f131341e-13f0-46a6-bf5a-54c35f749073\") " pod="opendatahub/model-serving-api-54d89bd5fc-qvlvs"
Sep 11 18:16:13.982746 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:13.982723    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f131341e-13f0-46a6-bf5a-54c35f749073-tls-certs\") pod \"model-serving-api-54d89bd5fc-qvlvs\" (UID: \"f131341e-13f0-46a6-bf5a-54c35f749073\") " pod="opendatahub/model-serving-api-54d89bd5fc-qvlvs"
Sep 11 18:16:14.266151 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:14.266112    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-qvlvs"
Sep 11 18:16:14.305101 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:14.305012    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rdq5p" event={"ID":"7f99264f-7197-4667-b258-a27550fb9ad1","Type":"ContainerStarted","Data":"fbf28d4353ae2dad2f9ae36bbb26e294e1a78b579dfb5b12cfb06a0d53abbd6d"}
Sep 11 18:16:14.545494 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:14.545319    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-qvlvs"]
Sep 11 18:16:14.548693 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:16:14.548660    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf131341e_13f0_46a6_bf5a_54c35f749073.slice/crio-56465fb0dc1bdb1302ef5e18a725061875bb7bb39823dc2dceca772df12d7795 WatchSource:0}: Error finding container 56465fb0dc1bdb1302ef5e18a725061875bb7bb39823dc2dceca772df12d7795: Status 404 returned error can't find the container with id 56465fb0dc1bdb1302ef5e18a725061875bb7bb39823dc2dceca772df12d7795
Sep 11 18:16:15.314293 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:15.314199    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-qvlvs" event={"ID":"f131341e-13f0-46a6-bf5a-54c35f749073","Type":"ContainerStarted","Data":"56465fb0dc1bdb1302ef5e18a725061875bb7bb39823dc2dceca772df12d7795"}
Sep 11 18:16:16.274286 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:16.274258    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6dd5f9dbb5-qb7nd"
Sep 11 18:16:18.328300 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:18.328260    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rdq5p" event={"ID":"7f99264f-7197-4667-b258-a27550fb9ad1","Type":"ContainerStarted","Data":"0ff5b96b1227b5facaa184639bc9d4c097db6ef20c67f2a60202fb42098acf9c"}
Sep 11 18:16:18.328757 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:18.328371    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-rdq5p"
Sep 11 18:16:18.329895 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:18.329871    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-qvlvs" event={"ID":"f131341e-13f0-46a6-bf5a-54c35f749073","Type":"ContainerStarted","Data":"43b1c443e52dae207a6e933ed4c73683ef78b98a0eedb1e49f2eeb2c109a297f"}
Sep 11 18:16:18.329996 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:18.329916    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-qvlvs"
Sep 11 18:16:18.352669 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:18.352601    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-rdq5p" podStartSLOduration=2.398049842 podStartE2EDuration="6.35256428s" totalImagesPullingTime="3.954514438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:18.351349949 +0000 UTC m=+376.952711127" watchObservedRunningTime="2026-09-11 18:16:18.35256428 +0000 UTC m=+376.953925444"
Sep 11 18:16:18.379425 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:18.379376    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-qvlvs" podStartSLOduration=2.052000513 podStartE2EDuration="5.379358774s" totalImagesPullingTime="3.327358261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:18.378572655 +0000 UTC m=+376.979933819" watchObservedRunningTime="2026-09-11 18:16:18.379358774 +0000 UTC m=+376.980719937"
Sep 11 18:16:24.043530 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:24.043474    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5xzh2"]
Sep 11 18:16:24.046984 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:24.046954    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5xzh2"
Sep 11 18:16:24.049506 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:24.049479    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 11 18:16:24.049641 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:24.049557    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-tfb66\""
Sep 11 18:16:24.059027 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:24.059001    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5xzh2"]
Sep 11 18:16:24.164105 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:24.164062    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msq9l\" (UniqueName: \"kubernetes.io/projected/80b74d21-4af5-4b9a-8164-03e70757b9fb-kube-api-access-msq9l\") pod \"llmisvc-controller-manager-777c6bd7d5-5xzh2\" (UID: \"80b74d21-4af5-4b9a-8164-03e70757b9fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5xzh2"
Sep 11 18:16:24.164267 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:24.164115    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80b74d21-4af5-4b9a-8164-03e70757b9fb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5xzh2\" (UID: \"80b74d21-4af5-4b9a-8164-03e70757b9fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5xzh2"
Sep 11 18:16:24.265006 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:24.264964    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msq9l\" (UniqueName: \"kubernetes.io/projected/80b74d21-4af5-4b9a-8164-03e70757b9fb-kube-api-access-msq9l\") pod \"llmisvc-controller-manager-777c6bd7d5-5xzh2\" (UID: \"80b74d21-4af5-4b9a-8164-03e70757b9fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5xzh2"
Sep 11 18:16:24.265006 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:24.265013    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80b74d21-4af5-4b9a-8164-03e70757b9fb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5xzh2\" (UID: \"80b74d21-4af5-4b9a-8164-03e70757b9fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5xzh2"
Sep 11 18:16:24.267652 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:24.267625    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80b74d21-4af5-4b9a-8164-03e70757b9fb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5xzh2\" (UID: \"80b74d21-4af5-4b9a-8164-03e70757b9fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5xzh2"
Sep 11 18:16:24.279511 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:24.279479    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msq9l\" (UniqueName: \"kubernetes.io/projected/80b74d21-4af5-4b9a-8164-03e70757b9fb-kube-api-access-msq9l\") pod \"llmisvc-controller-manager-777c6bd7d5-5xzh2\" (UID: \"80b74d21-4af5-4b9a-8164-03e70757b9fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5xzh2"
Sep 11 18:16:24.356161 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:24.356056    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5xzh2"
Sep 11 18:16:24.651683 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:24.651657    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5xzh2"]
Sep 11 18:16:24.654260 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:16:24.654230    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod80b74d21_4af5_4b9a_8164_03e70757b9fb.slice/crio-6fafa89037212990601491f8b802d97a90efe5c65a301594ed900560218f4f90 WatchSource:0}: Error finding container 6fafa89037212990601491f8b802d97a90efe5c65a301594ed900560218f4f90: Status 404 returned error can't find the container with id 6fafa89037212990601491f8b802d97a90efe5c65a301594ed900560218f4f90
Sep 11 18:16:25.357188 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:25.357145    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5xzh2" event={"ID":"80b74d21-4af5-4b9a-8164-03e70757b9fb","Type":"ContainerStarted","Data":"6fafa89037212990601491f8b802d97a90efe5c65a301594ed900560218f4f90"}
Sep 11 18:16:28.372885 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:28.372848    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5xzh2" event={"ID":"80b74d21-4af5-4b9a-8164-03e70757b9fb","Type":"ContainerStarted","Data":"8611fce8e57273c36f09cc2d151be70e142dccf738bed3998566ad0b175b817b"}
Sep 11 18:16:28.373314 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:28.372998    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5xzh2"
Sep 11 18:16:28.407332 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:28.407257    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5xzh2" podStartSLOduration=2.247085885 podStartE2EDuration="5.407238901s" totalImagesPullingTime="3.160153016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:28.40314604 +0000 UTC m=+387.004507204" watchObservedRunningTime="2026-09-11 18:16:28.407238901 +0000 UTC m=+387.008600063"
Sep 11 18:16:29.335694 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:29.335661    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-rdq5p"
Sep 11 18:16:29.337498 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:29.337473    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-qvlvs"
Sep 11 18:16:59.380135 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:16:59.380095    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5xzh2"
Sep 11 18:17:42.774152 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:42.774109    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-9szsj"]
Sep 11 18:17:42.776192 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:42.776175    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-9szsj"
Sep 11 18:17:42.779647 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:42.779626    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 18:17:42.779647 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:42.779643    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 18:17:42.779901 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:42.779885    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-qhfbj\""
Sep 11 18:17:42.791045 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:42.791022    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-9szsj"]
Sep 11 18:17:42.862606 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:42.862552    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7v97\" (UniqueName: \"kubernetes.io/projected/2408d02c-ebba-43f0-9470-872c6ffb1ea6-kube-api-access-m7v97\") pod \"authorino-operator-76b478bd78-9szsj\" (UID: \"2408d02c-ebba-43f0-9470-872c6ffb1ea6\") " pod="kuadrant-system/authorino-operator-76b478bd78-9szsj"
Sep 11 18:17:42.963472 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:42.963436    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7v97\" (UniqueName: \"kubernetes.io/projected/2408d02c-ebba-43f0-9470-872c6ffb1ea6-kube-api-access-m7v97\") pod \"authorino-operator-76b478bd78-9szsj\" (UID: \"2408d02c-ebba-43f0-9470-872c6ffb1ea6\") " pod="kuadrant-system/authorino-operator-76b478bd78-9szsj"
Sep 11 18:17:42.973925 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:42.973891    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7v97\" (UniqueName: \"kubernetes.io/projected/2408d02c-ebba-43f0-9470-872c6ffb1ea6-kube-api-access-m7v97\") pod \"authorino-operator-76b478bd78-9szsj\" (UID: \"2408d02c-ebba-43f0-9470-872c6ffb1ea6\") " pod="kuadrant-system/authorino-operator-76b478bd78-9szsj"
Sep 11 18:17:43.085947 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:43.085848    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-9szsj"
Sep 11 18:17:43.383022 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:43.382999    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-9szsj"]
Sep 11 18:17:43.384874 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:17:43.384848    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2408d02c_ebba_43f0_9470_872c6ffb1ea6.slice/crio-dae275555c07bddc48dda5f4310680e05fa73f0c8096f90eacfb7b634a290e09 WatchSource:0}: Error finding container dae275555c07bddc48dda5f4310680e05fa73f0c8096f90eacfb7b634a290e09: Status 404 returned error can't find the container with id dae275555c07bddc48dda5f4310680e05fa73f0c8096f90eacfb7b634a290e09
Sep 11 18:17:43.665448 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:43.665361    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-9szsj" event={"ID":"2408d02c-ebba-43f0-9470-872c6ffb1ea6","Type":"ContainerStarted","Data":"dae275555c07bddc48dda5f4310680e05fa73f0c8096f90eacfb7b634a290e09"}
Sep 11 18:17:47.683417 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:47.683383    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-9szsj" event={"ID":"2408d02c-ebba-43f0-9470-872c6ffb1ea6","Type":"ContainerStarted","Data":"6e8677988bdbab93e421647f3351698938cceb5d7457b061261bf87219e602f7"}
Sep 11 18:17:47.683856 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:47.683502    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-9szsj"
Sep 11 18:17:47.704685 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:47.704632    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-9szsj" podStartSLOduration=2.431299049 podStartE2EDuration="5.704613705s" totalImagesPullingTime="3.273314656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:47.70262541 +0000 UTC m=+466.303986577" watchObservedRunningTime="2026-09-11 18:17:47.704613705 +0000 UTC m=+466.305974864"
Sep 11 18:17:57.503273 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:57.503240    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrgfh"]
Sep 11 18:17:57.511142 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:57.511113    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrgfh"
Sep 11 18:17:57.513857 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:57.513837    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 11 18:17:57.514059 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:57.514044    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-p2rph\""
Sep 11 18:17:57.514195 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:57.514174    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 11 18:17:57.519826 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:57.519803    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrgfh"]
Sep 11 18:17:57.582563 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:57.582523    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f7d610e7-ead8-4b05-ba09-95d6f66609e1-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-nrgfh\" (UID: \"f7d610e7-ead8-4b05-ba09-95d6f66609e1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrgfh"
Sep 11 18:17:57.582563 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:57.582568    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f7d610e7-ead8-4b05-ba09-95d6f66609e1-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-nrgfh\" (UID: \"f7d610e7-ead8-4b05-ba09-95d6f66609e1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrgfh"
Sep 11 18:17:57.582795 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:57.582615    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqrxm\" (UniqueName: \"kubernetes.io/projected/f7d610e7-ead8-4b05-ba09-95d6f66609e1-kube-api-access-hqrxm\") pod \"kuadrant-console-plugin-fbdd4f445-nrgfh\" (UID: \"f7d610e7-ead8-4b05-ba09-95d6f66609e1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrgfh"
Sep 11 18:17:57.683724 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:57.683674    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f7d610e7-ead8-4b05-ba09-95d6f66609e1-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-nrgfh\" (UID: \"f7d610e7-ead8-4b05-ba09-95d6f66609e1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrgfh"
Sep 11 18:17:57.683833 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:57.683731    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqrxm\" (UniqueName: \"kubernetes.io/projected/f7d610e7-ead8-4b05-ba09-95d6f66609e1-kube-api-access-hqrxm\") pod \"kuadrant-console-plugin-fbdd4f445-nrgfh\" (UID: \"f7d610e7-ead8-4b05-ba09-95d6f66609e1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrgfh"
Sep 11 18:17:57.683833 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:57.683791    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f7d610e7-ead8-4b05-ba09-95d6f66609e1-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-nrgfh\" (UID: \"f7d610e7-ead8-4b05-ba09-95d6f66609e1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrgfh"
Sep 11 18:17:57.683905 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:17:57.683891    2576 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 11 18:17:57.683976 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:17:57.683965    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7d610e7-ead8-4b05-ba09-95d6f66609e1-plugin-serving-cert podName:f7d610e7-ead8-4b05-ba09-95d6f66609e1 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:58.183946315 +0000 UTC m=+476.785307462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/f7d610e7-ead8-4b05-ba09-95d6f66609e1-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-nrgfh" (UID: "f7d610e7-ead8-4b05-ba09-95d6f66609e1") : secret "plugin-serving-cert" not found
Sep 11 18:17:57.684357 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:57.684336    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f7d610e7-ead8-4b05-ba09-95d6f66609e1-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-nrgfh\" (UID: \"f7d610e7-ead8-4b05-ba09-95d6f66609e1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrgfh"
Sep 11 18:17:57.696414 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:57.696381    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqrxm\" (UniqueName: \"kubernetes.io/projected/f7d610e7-ead8-4b05-ba09-95d6f66609e1-kube-api-access-hqrxm\") pod \"kuadrant-console-plugin-fbdd4f445-nrgfh\" (UID: \"f7d610e7-ead8-4b05-ba09-95d6f66609e1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrgfh"
Sep 11 18:17:58.188634 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:58.188568    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f7d610e7-ead8-4b05-ba09-95d6f66609e1-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-nrgfh\" (UID: \"f7d610e7-ead8-4b05-ba09-95d6f66609e1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrgfh"
Sep 11 18:17:58.191048 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:58.191015    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f7d610e7-ead8-4b05-ba09-95d6f66609e1-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-nrgfh\" (UID: \"f7d610e7-ead8-4b05-ba09-95d6f66609e1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrgfh"
Sep 11 18:17:58.420461 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:58.420405    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrgfh"
Sep 11 18:17:58.690010 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:58.689942    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-9szsj"
Sep 11 18:17:58.692258 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:58.692223    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrgfh"]
Sep 11 18:17:58.695502 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:17:58.695478    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7d610e7_ead8_4b05_ba09_95d6f66609e1.slice/crio-3b9d56ccb8fc79accb0b1ec264910476b137a1b9687a7f4ffb7cc44af8275c09 WatchSource:0}: Error finding container 3b9d56ccb8fc79accb0b1ec264910476b137a1b9687a7f4ffb7cc44af8275c09: Status 404 returned error can't find the container with id 3b9d56ccb8fc79accb0b1ec264910476b137a1b9687a7f4ffb7cc44af8275c09
Sep 11 18:17:58.726200 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:17:58.726124    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrgfh" event={"ID":"f7d610e7-ead8-4b05-ba09-95d6f66609e1","Type":"ContainerStarted","Data":"3b9d56ccb8fc79accb0b1ec264910476b137a1b9687a7f4ffb7cc44af8275c09"}
Sep 11 18:18:03.753690 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:18:03.753651    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrgfh" event={"ID":"f7d610e7-ead8-4b05-ba09-95d6f66609e1","Type":"ContainerStarted","Data":"beeb9da4bdeb08351e8a6ee49954438a996353765719f7cb2b46bb0f4f720770"}
Sep 11 18:18:03.770913 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:18:03.770865    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrgfh" podStartSLOduration=2.040241211 podStartE2EDuration="6.770849586s" totalImagesPullingTime="4.730608375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:03.770181981 +0000 UTC m=+482.371543141" watchObservedRunningTime="2026-09-11 18:18:03.770849586 +0000 UTC m=+482.372210749"
Sep 11 18:18:55.975255 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:18:55.975178    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:18:55.978259 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:18:55.978242    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"
Sep 11 18:18:55.980752 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:18:55.980729    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 18:18:55.986656 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:18:55.986630    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:18:55.998330 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:18:55.998299    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5b1c0c2f-4b2c-47e2-b0b6-cda9d763b33d-config-file\") pod \"limitador-limitador-84994f7ffc-b4kzn\" (UID: \"5b1c0c2f-4b2c-47e2-b0b6-cda9d763b33d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"
Sep 11 18:18:55.998479 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:18:55.998348    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj7vj\" (UniqueName: \"kubernetes.io/projected/5b1c0c2f-4b2c-47e2-b0b6-cda9d763b33d-kube-api-access-gj7vj\") pod \"limitador-limitador-84994f7ffc-b4kzn\" (UID: \"5b1c0c2f-4b2c-47e2-b0b6-cda9d763b33d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"
Sep 11 18:18:56.000414 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:18:56.000389    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:18:56.099488 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:18:56.099447    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5b1c0c2f-4b2c-47e2-b0b6-cda9d763b33d-config-file\") pod \"limitador-limitador-84994f7ffc-b4kzn\" (UID: \"5b1c0c2f-4b2c-47e2-b0b6-cda9d763b33d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"
Sep 11 18:18:56.099699 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:18:56.099508    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj7vj\" (UniqueName: \"kubernetes.io/projected/5b1c0c2f-4b2c-47e2-b0b6-cda9d763b33d-kube-api-access-gj7vj\") pod \"limitador-limitador-84994f7ffc-b4kzn\" (UID: \"5b1c0c2f-4b2c-47e2-b0b6-cda9d763b33d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"
Sep 11 18:18:56.100097 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:18:56.100077    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5b1c0c2f-4b2c-47e2-b0b6-cda9d763b33d-config-file\") pod \"limitador-limitador-84994f7ffc-b4kzn\" (UID: \"5b1c0c2f-4b2c-47e2-b0b6-cda9d763b33d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"
Sep 11 18:18:56.108056 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:18:56.108026    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj7vj\" (UniqueName: \"kubernetes.io/projected/5b1c0c2f-4b2c-47e2-b0b6-cda9d763b33d-kube-api-access-gj7vj\") pod \"limitador-limitador-84994f7ffc-b4kzn\" (UID: \"5b1c0c2f-4b2c-47e2-b0b6-cda9d763b33d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"
Sep 11 18:18:56.289121 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:18:56.289025    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"
Sep 11 18:18:56.566347 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:18:56.566322    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:18:56.569051 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:18:56.569021    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1c0c2f_4b2c_47e2_b0b6_cda9d763b33d.slice/crio-fb959a31c53f3124e1e82db1f7e6ad8e562fe02757716a53a7150a14527a6a55 WatchSource:0}: Error finding container fb959a31c53f3124e1e82db1f7e6ad8e562fe02757716a53a7150a14527a6a55: Status 404 returned error can't find the container with id fb959a31c53f3124e1e82db1f7e6ad8e562fe02757716a53a7150a14527a6a55
Sep 11 18:18:56.959696 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:18:56.959662    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-b4kzn" event={"ID":"5b1c0c2f-4b2c-47e2-b0b6-cda9d763b33d","Type":"ContainerStarted","Data":"fb959a31c53f3124e1e82db1f7e6ad8e562fe02757716a53a7150a14527a6a55"}
Sep 11 18:18:57.965015 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:18:57.964972    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-b4kzn" event={"ID":"5b1c0c2f-4b2c-47e2-b0b6-cda9d763b33d","Type":"ContainerStarted","Data":"b8534764a359b1e0ab533f8443fe7f9526dbbcba8b300b2154caa5aaaa8ae3f3"}
Sep 11 18:18:57.965425 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:18:57.965168    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"
Sep 11 18:18:57.984679 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:18:57.984624    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-b4kzn" podStartSLOduration=2.15335659 podStartE2EDuration="2.984605306s" totalImagesPullingTime="831.248716ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:57.983189758 +0000 UTC m=+536.584550923" watchObservedRunningTime="2026-09-11 18:18:57.984605306 +0000 UTC m=+536.585966466"
Sep 11 18:19:08.971037 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:19:08.971002    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"
Sep 11 18:20:09.117442 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:20:09.117409    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:20:33.323284 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:20:33.323205    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:20:48.118994 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:20:48.118957    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:21:05.096809 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.096769    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"]
Sep 11 18:21:05.099897 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.099880    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"
Sep 11 18:21:05.103202 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.103179    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 11 18:21:05.103202 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.103181    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 18:21:05.103397 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.103209    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-qdfkl\""
Sep 11 18:21:05.103397 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.103251    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 18:21:05.109869 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.109849    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"]
Sep 11 18:21:05.162333 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.162302    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cef9934-b9ad-4ce6-98b3-c268693fed89-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz\" (UID: \"5cef9934-b9ad-4ce6-98b3-c268693fed89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"
Sep 11 18:21:05.162333 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.162333    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cef9934-b9ad-4ce6-98b3-c268693fed89-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz\" (UID: \"5cef9934-b9ad-4ce6-98b3-c268693fed89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"
Sep 11 18:21:05.162525 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.162406    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cef9934-b9ad-4ce6-98b3-c268693fed89-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz\" (UID: \"5cef9934-b9ad-4ce6-98b3-c268693fed89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"
Sep 11 18:21:05.162525 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.162421    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cef9934-b9ad-4ce6-98b3-c268693fed89-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz\" (UID: \"5cef9934-b9ad-4ce6-98b3-c268693fed89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"
Sep 11 18:21:05.162525 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.162437    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2xvp\" (UniqueName: \"kubernetes.io/projected/5cef9934-b9ad-4ce6-98b3-c268693fed89-kube-api-access-c2xvp\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz\" (UID: \"5cef9934-b9ad-4ce6-98b3-c268693fed89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"
Sep 11 18:21:05.162525 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.162473    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cef9934-b9ad-4ce6-98b3-c268693fed89-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz\" (UID: \"5cef9934-b9ad-4ce6-98b3-c268693fed89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"
Sep 11 18:21:05.263875 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.263831    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cef9934-b9ad-4ce6-98b3-c268693fed89-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz\" (UID: \"5cef9934-b9ad-4ce6-98b3-c268693fed89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"
Sep 11 18:21:05.263875 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.263880    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cef9934-b9ad-4ce6-98b3-c268693fed89-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz\" (UID: \"5cef9934-b9ad-4ce6-98b3-c268693fed89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"
Sep 11 18:21:05.264083 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.264001    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cef9934-b9ad-4ce6-98b3-c268693fed89-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz\" (UID: \"5cef9934-b9ad-4ce6-98b3-c268693fed89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"
Sep 11 18:21:05.264139 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.264120    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cef9934-b9ad-4ce6-98b3-c268693fed89-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz\" (UID: \"5cef9934-b9ad-4ce6-98b3-c268693fed89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"
Sep 11 18:21:05.264185 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.264155    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cef9934-b9ad-4ce6-98b3-c268693fed89-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz\" (UID: \"5cef9934-b9ad-4ce6-98b3-c268693fed89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"
Sep 11 18:21:05.264237 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.264181    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2xvp\" (UniqueName: \"kubernetes.io/projected/5cef9934-b9ad-4ce6-98b3-c268693fed89-kube-api-access-c2xvp\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz\" (UID: \"5cef9934-b9ad-4ce6-98b3-c268693fed89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"
Sep 11 18:21:05.264314 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.264291    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cef9934-b9ad-4ce6-98b3-c268693fed89-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz\" (UID: \"5cef9934-b9ad-4ce6-98b3-c268693fed89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"
Sep 11 18:21:05.264478 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.264457    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cef9934-b9ad-4ce6-98b3-c268693fed89-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz\" (UID: \"5cef9934-b9ad-4ce6-98b3-c268693fed89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"
Sep 11 18:21:05.264544 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.264505    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cef9934-b9ad-4ce6-98b3-c268693fed89-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz\" (UID: \"5cef9934-b9ad-4ce6-98b3-c268693fed89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"
Sep 11 18:21:05.266138 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.266121    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cef9934-b9ad-4ce6-98b3-c268693fed89-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz\" (UID: \"5cef9934-b9ad-4ce6-98b3-c268693fed89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"
Sep 11 18:21:05.266524 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.266504    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cef9934-b9ad-4ce6-98b3-c268693fed89-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz\" (UID: \"5cef9934-b9ad-4ce6-98b3-c268693fed89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"
Sep 11 18:21:05.272248 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.272226    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2xvp\" (UniqueName: \"kubernetes.io/projected/5cef9934-b9ad-4ce6-98b3-c268693fed89-kube-api-access-c2xvp\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz\" (UID: \"5cef9934-b9ad-4ce6-98b3-c268693fed89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"
Sep 11 18:21:05.410457 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.410360    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"
Sep 11 18:21:05.602252 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.602217    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:21:05.696370 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.696343    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"]
Sep 11 18:21:05.698665 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:21:05.698638    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cef9934_b9ad_4ce6_98b3_c268693fed89.slice/crio-1c7e140d0c6f2188e37b9403f0cdd7e62ffe822e59252c3b987e4b5f7b176d64 WatchSource:0}: Error finding container 1c7e140d0c6f2188e37b9403f0cdd7e62ffe822e59252c3b987e4b5f7b176d64: Status 404 returned error can't find the container with id 1c7e140d0c6f2188e37b9403f0cdd7e62ffe822e59252c3b987e4b5f7b176d64
Sep 11 18:21:05.700101 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:05.700084    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:21:06.485323 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:06.485280    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz" event={"ID":"5cef9934-b9ad-4ce6-98b3-c268693fed89","Type":"ContainerStarted","Data":"1c7e140d0c6f2188e37b9403f0cdd7e62ffe822e59252c3b987e4b5f7b176d64"}
Sep 11 18:21:07.490491 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:07.490452    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz" event={"ID":"5cef9934-b9ad-4ce6-98b3-c268693fed89","Type":"ContainerStarted","Data":"42f3afd7b6d2b4efe1725f3d52487cdbde32067c479ce1e9f9d88a28a35c85fe"}
Sep 11 18:21:07.511377 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:07.511322    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz" podStartSLOduration=1.495270302 podStartE2EDuration="2.511306813s" totalImagesPullingTime="1.016036511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:21:07.509558298 +0000 UTC m=+666.110919492" watchObservedRunningTime="2026-09-11 18:21:07.511306813 +0000 UTC m=+666.112667975"
Sep 11 18:21:15.411075 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:15.411033    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"
Sep 11 18:21:15.411571 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:15.411104    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"
Sep 11 18:21:15.423348 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:15.423322    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"
Sep 11 18:21:15.534044 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:15.534009    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz"
Sep 11 18:21:17.611410 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:17.611374    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:21:29.614661 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:29.614626    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:21:39.996898 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:39.996861    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"]
Sep 11 18:21:40.000036 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:40.000013    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"
Sep 11 18:21:40.002262 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:40.002238    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 11 18:21:40.009341 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:40.009315    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"]
Sep 11 18:21:40.048478 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:40.048439    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1648fc00-a5bf-4da2-b76b-49dd4b239b84-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jxrxf\" (UID: \"1648fc00-a5bf-4da2-b76b-49dd4b239b84\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"
Sep 11 18:21:40.048676 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:40.048483    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1648fc00-a5bf-4da2-b76b-49dd4b239b84-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jxrxf\" (UID: \"1648fc00-a5bf-4da2-b76b-49dd4b239b84\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"
Sep 11 18:21:40.048676 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:40.048503    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1648fc00-a5bf-4da2-b76b-49dd4b239b84-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jxrxf\" (UID: \"1648fc00-a5bf-4da2-b76b-49dd4b239b84\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"
Sep 11 18:21:40.048676 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:40.048567    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n92rp\" (UniqueName: \"kubernetes.io/projected/1648fc00-a5bf-4da2-b76b-49dd4b239b84-kube-api-access-n92rp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jxrxf\" (UID: \"1648fc00-a5bf-4da2-b76b-49dd4b239b84\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"
Sep 11 18:21:40.048676 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:40.048623    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1648fc00-a5bf-4da2-b76b-49dd4b239b84-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jxrxf\" (UID: \"1648fc00-a5bf-4da2-b76b-49dd4b239b84\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"
Sep 11 18:21:40.048873 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:40.048676    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1648fc00-a5bf-4da2-b76b-49dd4b239b84-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jxrxf\" (UID: \"1648fc00-a5bf-4da2-b76b-49dd4b239b84\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"
Sep 11 18:21:40.150076 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:40.150031    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1648fc00-a5bf-4da2-b76b-49dd4b239b84-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jxrxf\" (UID: \"1648fc00-a5bf-4da2-b76b-49dd4b239b84\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"
Sep 11 18:21:40.150076 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:40.150077    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1648fc00-a5bf-4da2-b76b-49dd4b239b84-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jxrxf\" (UID: \"1648fc00-a5bf-4da2-b76b-49dd4b239b84\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"
Sep 11 18:21:40.150313 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:40.150093    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1648fc00-a5bf-4da2-b76b-49dd4b239b84-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jxrxf\" (UID: \"1648fc00-a5bf-4da2-b76b-49dd4b239b84\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"
Sep 11 18:21:40.150313 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:40.150128    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n92rp\" (UniqueName: \"kubernetes.io/projected/1648fc00-a5bf-4da2-b76b-49dd4b239b84-kube-api-access-n92rp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jxrxf\" (UID: \"1648fc00-a5bf-4da2-b76b-49dd4b239b84\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"
Sep 11 18:21:40.150313 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:40.150158    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1648fc00-a5bf-4da2-b76b-49dd4b239b84-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jxrxf\" (UID: \"1648fc00-a5bf-4da2-b76b-49dd4b239b84\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"
Sep 11 18:21:40.150313 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:40.150212    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1648fc00-a5bf-4da2-b76b-49dd4b239b84-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jxrxf\" (UID: \"1648fc00-a5bf-4da2-b76b-49dd4b239b84\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"
Sep 11 18:21:40.150621 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:40.150559    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1648fc00-a5bf-4da2-b76b-49dd4b239b84-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jxrxf\" (UID: \"1648fc00-a5bf-4da2-b76b-49dd4b239b84\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"
Sep 11 18:21:40.150725 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:40.150624    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1648fc00-a5bf-4da2-b76b-49dd4b239b84-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jxrxf\" (UID: \"1648fc00-a5bf-4da2-b76b-49dd4b239b84\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"
Sep 11 18:21:40.150725 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:40.150646    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1648fc00-a5bf-4da2-b76b-49dd4b239b84-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jxrxf\" (UID: \"1648fc00-a5bf-4da2-b76b-49dd4b239b84\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"
Sep 11 18:21:40.152358 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:40.152323    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1648fc00-a5bf-4da2-b76b-49dd4b239b84-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jxrxf\" (UID: \"1648fc00-a5bf-4da2-b76b-49dd4b239b84\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"
Sep 11 18:21:40.152694 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:40.152675    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1648fc00-a5bf-4da2-b76b-49dd4b239b84-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jxrxf\" (UID: \"1648fc00-a5bf-4da2-b76b-49dd4b239b84\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"
Sep 11 18:21:40.158619 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:40.158571    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n92rp\" (UniqueName: \"kubernetes.io/projected/1648fc00-a5bf-4da2-b76b-49dd4b239b84-kube-api-access-n92rp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jxrxf\" (UID: \"1648fc00-a5bf-4da2-b76b-49dd4b239b84\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"
Sep 11 18:21:40.310309 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:40.310224    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"
Sep 11 18:21:40.580899 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:40.580810    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"]
Sep 11 18:21:40.583905 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:21:40.583877    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1648fc00_a5bf_4da2_b76b_49dd4b239b84.slice/crio-4267b079c1601989c00963ce3410335280c6f83a49da0dbf9f22a8d29a68008e WatchSource:0}: Error finding container 4267b079c1601989c00963ce3410335280c6f83a49da0dbf9f22a8d29a68008e: Status 404 returned error can't find the container with id 4267b079c1601989c00963ce3410335280c6f83a49da0dbf9f22a8d29a68008e
Sep 11 18:21:40.628050 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:40.628013    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf" event={"ID":"1648fc00-a5bf-4da2-b76b-49dd4b239b84","Type":"ContainerStarted","Data":"4267b079c1601989c00963ce3410335280c6f83a49da0dbf9f22a8d29a68008e"}
Sep 11 18:21:41.633676 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:41.633631    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf" event={"ID":"1648fc00-a5bf-4da2-b76b-49dd4b239b84","Type":"ContainerStarted","Data":"94ee46cc4e991de8929590648c8e301603a8c97d509dd0c6439c9bba75da9c9f"}
Sep 11 18:21:41.651352 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:41.651296    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf" podStartSLOduration=2.473476283 podStartE2EDuration="2.651278447s" totalImagesPullingTime="177.802164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:21:41.650114968 +0000 UTC m=+700.251476132" watchObservedRunningTime="2026-09-11 18:21:41.651278447 +0000 UTC m=+700.252639604"
Sep 11 18:21:41.910433 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:41.910354    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:21:50.311200 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:50.311165    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"
Sep 11 18:21:50.311613 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:50.311230    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"
Sep 11 18:21:50.323967 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:50.323938    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"
Sep 11 18:21:50.682621 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:50.682518    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jxrxf"
Sep 11 18:21:51.811345 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:21:51.811304    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:23:25.643845 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:25.643757    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7851-f68ca2"]
Sep 11 18:23:25.647156 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:25.647130    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-f68ca2"
Sep 11 18:23:25.649491 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:25.649458    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fz6cx\""
Sep 11 18:23:25.652731 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:25.652704    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7851-f68ca2"]
Sep 11 18:23:25.727893 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:25.727856    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8pvt\" (UniqueName: \"kubernetes.io/projected/1ea67197-a088-44d9-ab49-156a437ef12f-kube-api-access-q8pvt\") pod \"test-curl-7851-f68ca2\" (UID: \"1ea67197-a088-44d9-ab49-156a437ef12f\") " pod="opendatahub/test-curl-7851-f68ca2"
Sep 11 18:23:25.829144 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:25.829108    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8pvt\" (UniqueName: \"kubernetes.io/projected/1ea67197-a088-44d9-ab49-156a437ef12f-kube-api-access-q8pvt\") pod \"test-curl-7851-f68ca2\" (UID: \"1ea67197-a088-44d9-ab49-156a437ef12f\") " pod="opendatahub/test-curl-7851-f68ca2"
Sep 11 18:23:25.837251 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:25.837222    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8pvt\" (UniqueName: \"kubernetes.io/projected/1ea67197-a088-44d9-ab49-156a437ef12f-kube-api-access-q8pvt\") pod \"test-curl-7851-f68ca2\" (UID: \"1ea67197-a088-44d9-ab49-156a437ef12f\") " pod="opendatahub/test-curl-7851-f68ca2"
Sep 11 18:23:25.957260 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:25.957222    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-f68ca2"
Sep 11 18:23:26.224549 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:26.224477    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7851-f68ca2"]
Sep 11 18:23:26.227188 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:23:26.227159    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ea67197_a088_44d9_ab49_156a437ef12f.slice/crio-60f6368eb1f0660183263c6e7b1c0b73822dbf94f340fabd3f26d436b3623dcb WatchSource:0}: Error finding container 60f6368eb1f0660183263c6e7b1c0b73822dbf94f340fabd3f26d436b3623dcb: Status 404 returned error can't find the container with id 60f6368eb1f0660183263c6e7b1c0b73822dbf94f340fabd3f26d436b3623dcb
Sep 11 18:23:27.074001 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:27.073967    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7851-f68ca2" event={"ID":"1ea67197-a088-44d9-ab49-156a437ef12f","Type":"ContainerStarted","Data":"60f6368eb1f0660183263c6e7b1c0b73822dbf94f340fabd3f26d436b3623dcb"}
Sep 11 18:23:27.669512 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:27.669470    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:23:28.080523 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:28.080482    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7851-f68ca2" event={"ID":"1ea67197-a088-44d9-ab49-156a437ef12f","Type":"ContainerStarted","Data":"e5e684ad2c1e0439e9f7fb80d4de9b4e164f5be33a946fcff2e97133ad409d38"}
Sep 11 18:23:28.672405 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:28.672343    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7851-f68ca2" podStartSLOduration=2.741831856 podStartE2EDuration="3.672323321s" totalImagesPullingTime="930.491465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:28.096225168 +0000 UTC m=+806.697586313" watchObservedRunningTime="2026-09-11 18:23:28.672323321 +0000 UTC m=+807.273684485"
Sep 11 18:23:28.672623 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:28.672570    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7851-f68ca2"]
Sep 11 18:23:28.678856 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:28.678823    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7851-f68ca2"]
Sep 11 18:23:28.805931 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:28.805845    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7851-88c088"]
Sep 11 18:23:28.806148 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:28.806135    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ea67197-a088-44d9-ab49-156a437ef12f" containerName="test-curl-7851-f68ca2"
Sep 11 18:23:28.806195 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:28.806149    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ea67197-a088-44d9-ab49-156a437ef12f" containerName="test-curl-7851-f68ca2"
Sep 11 18:23:28.806232 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:28.806201    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ea67197-a088-44d9-ab49-156a437ef12f" containerName="test-curl-7851-f68ca2"
Sep 11 18:23:28.809017 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:28.808999    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-88c088"
Sep 11 18:23:28.813355 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:28.813326    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7851-88c088"]
Sep 11 18:23:28.849937 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:28.849899    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpx5z\" (UniqueName: \"kubernetes.io/projected/c51afc09-c4e6-44be-9a3d-ddefddab2370-kube-api-access-qpx5z\") pod \"test-curl-7851-88c088\" (UID: \"c51afc09-c4e6-44be-9a3d-ddefddab2370\") " pod="opendatahub/test-curl-7851-88c088"
Sep 11 18:23:28.950325 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:28.950284    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpx5z\" (UniqueName: \"kubernetes.io/projected/c51afc09-c4e6-44be-9a3d-ddefddab2370-kube-api-access-qpx5z\") pod \"test-curl-7851-88c088\" (UID: \"c51afc09-c4e6-44be-9a3d-ddefddab2370\") " pod="opendatahub/test-curl-7851-88c088"
Sep 11 18:23:28.958767 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:28.958734    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpx5z\" (UniqueName: \"kubernetes.io/projected/c51afc09-c4e6-44be-9a3d-ddefddab2370-kube-api-access-qpx5z\") pod \"test-curl-7851-88c088\" (UID: \"c51afc09-c4e6-44be-9a3d-ddefddab2370\") " pod="opendatahub/test-curl-7851-88c088"
Sep 11 18:23:29.085136 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:23:29.085059    2576 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7851-f68ca2" container="test-curl-7851-f68ca2"
Sep 11 18:23:29.119365 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.119329    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-88c088"
Sep 11 18:23:29.174623 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.174569    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2"]
Sep 11 18:23:29.178832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.178807    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2"
Sep 11 18:23:29.184617 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.184477    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-d73031-dockercfg-plwbn\""
Sep 11 18:23:29.184617 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.184611    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-d73031\""
Sep 11 18:23:29.185203 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.185120    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 18:23:29.190168 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.187934    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 18:23:29.198467 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.194818    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2"]
Sep 11 18:23:29.253454 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.253414    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13435514-2ef4-401b-b2d6-cfe8fd0f52ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2\" (UID: \"13435514-2ef4-401b-b2d6-cfe8fd0f52ef\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2"
Sep 11 18:23:29.253636 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.253467    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljj5g\" (UniqueName: \"kubernetes.io/projected/13435514-2ef4-401b-b2d6-cfe8fd0f52ef-kube-api-access-ljj5g\") pod \"payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2\" (UID: \"13435514-2ef4-401b-b2d6-cfe8fd0f52ef\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2"
Sep 11 18:23:29.354204 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.354118    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13435514-2ef4-401b-b2d6-cfe8fd0f52ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2\" (UID: \"13435514-2ef4-401b-b2d6-cfe8fd0f52ef\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2"
Sep 11 18:23:29.354204 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.354173    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljj5g\" (UniqueName: \"kubernetes.io/projected/13435514-2ef4-401b-b2d6-cfe8fd0f52ef-kube-api-access-ljj5g\") pod \"payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2\" (UID: \"13435514-2ef4-401b-b2d6-cfe8fd0f52ef\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2"
Sep 11 18:23:29.355019 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.354990    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13435514-2ef4-401b-b2d6-cfe8fd0f52ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2\" (UID: \"13435514-2ef4-401b-b2d6-cfe8fd0f52ef\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2"
Sep 11 18:23:29.365866 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.365824    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljj5g\" (UniqueName: \"kubernetes.io/projected/13435514-2ef4-401b-b2d6-cfe8fd0f52ef-kube-api-access-ljj5g\") pod \"payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2\" (UID: \"13435514-2ef4-401b-b2d6-cfe8fd0f52ef\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2"
Sep 11 18:23:29.391263 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.390937    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7851-88c088"]
Sep 11 18:23:29.393881 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:23:29.393853    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc51afc09_c4e6_44be_9a3d_ddefddab2370.slice/crio-e5d2989cf50c8f3bcc600e5a98533e347eef0ddaa3265a0b5e41f40c53ef37ee WatchSource:0}: Error finding container e5d2989cf50c8f3bcc600e5a98533e347eef0ddaa3265a0b5e41f40c53ef37ee: Status 404 returned error can't find the container with id e5d2989cf50c8f3bcc600e5a98533e347eef0ddaa3265a0b5e41f40c53ef37ee
Sep 11 18:23:29.501383 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.501344    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2"
Sep 11 18:23:29.582795 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.582759    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m"]
Sep 11 18:23:29.586180 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.586155    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m"
Sep 11 18:23:29.588504 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.588479    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:23:29.588504 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.588495    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 18:23:29.588861 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.588816    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-553596-serving-cert\""
Sep 11 18:23:29.588956 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.588865    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 18:23:29.588956 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.588842    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bqwkv\""
Sep 11 18:23:29.596917 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.596888    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m"]
Sep 11 18:23:29.655327 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.655232    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22-metrics-certs\") pod \"maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m\" (UID: \"ca9cd8a2-5d2a-4b6e-a016-f771ca54da22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m"
Sep 11 18:23:29.655327 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.655291    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m\" (UID: \"ca9cd8a2-5d2a-4b6e-a016-f771ca54da22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m"
Sep 11 18:23:29.655327 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.655328    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22-maas-api-tls\") pod \"maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m\" (UID: \"ca9cd8a2-5d2a-4b6e-a016-f771ca54da22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m"
Sep 11 18:23:29.655558 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.655368    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p6h9\" (UniqueName: \"kubernetes.io/projected/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22-kube-api-access-7p6h9\") pod \"maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m\" (UID: \"ca9cd8a2-5d2a-4b6e-a016-f771ca54da22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m"
Sep 11 18:23:29.755900 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.755859    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22-metrics-certs\") pod \"maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m\" (UID: \"ca9cd8a2-5d2a-4b6e-a016-f771ca54da22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m"
Sep 11 18:23:29.756065 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.755926    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m\" (UID: \"ca9cd8a2-5d2a-4b6e-a016-f771ca54da22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m"
Sep 11 18:23:29.756065 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.755964    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22-maas-api-tls\") pod \"maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m\" (UID: \"ca9cd8a2-5d2a-4b6e-a016-f771ca54da22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m"
Sep 11 18:23:29.756065 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.756005    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p6h9\" (UniqueName: \"kubernetes.io/projected/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22-kube-api-access-7p6h9\") pod \"maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m\" (UID: \"ca9cd8a2-5d2a-4b6e-a016-f771ca54da22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m"
Sep 11 18:23:29.756811 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.756773    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m\" (UID: \"ca9cd8a2-5d2a-4b6e-a016-f771ca54da22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m"
Sep 11 18:23:29.758490 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.758469    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22-maas-api-tls\") pod \"maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m\" (UID: \"ca9cd8a2-5d2a-4b6e-a016-f771ca54da22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m"
Sep 11 18:23:29.758832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.758816    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22-metrics-certs\") pod \"maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m\" (UID: \"ca9cd8a2-5d2a-4b6e-a016-f771ca54da22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m"
Sep 11 18:23:29.764213 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.764192    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p6h9\" (UniqueName: \"kubernetes.io/projected/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22-kube-api-access-7p6h9\") pod \"maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m\" (UID: \"ca9cd8a2-5d2a-4b6e-a016-f771ca54da22\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m"
Sep 11 18:23:29.780163 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.780133    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2"]
Sep 11 18:23:29.782793 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:23:29.782769    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13435514_2ef4_401b_b2d6_cfe8fd0f52ef.slice/crio-6818c90478e7732975cf2cddda5f7ba4ea25fbef68fd75c10a505042e59f4c73 WatchSource:0}: Error finding container 6818c90478e7732975cf2cddda5f7ba4ea25fbef68fd75c10a505042e59f4c73: Status 404 returned error can't find the container with id 6818c90478e7732975cf2cddda5f7ba4ea25fbef68fd75c10a505042e59f4c73
Sep 11 18:23:29.898812 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:29.898764    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m"
Sep 11 18:23:30.089968 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:30.089926    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2" event={"ID":"13435514-2ef4-401b-b2d6-cfe8fd0f52ef","Type":"ContainerStarted","Data":"6818c90478e7732975cf2cddda5f7ba4ea25fbef68fd75c10a505042e59f4c73"}
Sep 11 18:23:30.091509 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:30.091480    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7851-88c088" event={"ID":"c51afc09-c4e6-44be-9a3d-ddefddab2370","Type":"ContainerStarted","Data":"0c3efd3a3097de66e4dc7257492ce1920fa0139a3057505068780726356633aa"}
Sep 11 18:23:30.091672 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:30.091514    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7851-88c088" event={"ID":"c51afc09-c4e6-44be-9a3d-ddefddab2370","Type":"ContainerStarted","Data":"e5d2989cf50c8f3bcc600e5a98533e347eef0ddaa3265a0b5e41f40c53ef37ee"}
Sep 11 18:23:30.091738 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:30.091682    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7851-f68ca2" podUID="1ea67197-a088-44d9-ab49-156a437ef12f" containerName="test-curl-7851-f68ca2" containerID="cri-o://e5e684ad2c1e0439e9f7fb80d4de9b4e164f5be33a946fcff2e97133ad409d38" gracePeriod=2
Sep 11 18:23:30.110933 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:30.110884    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7851-88c088" podStartSLOduration=2.110867347 podStartE2EDuration="2.110867347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:30.109842049 +0000 UTC m=+808.711203212" watchObservedRunningTime="2026-09-11 18:23:30.110867347 +0000 UTC m=+808.712228507"
Sep 11 18:23:30.178333 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:30.178225    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m"]
Sep 11 18:23:30.181732 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:23:30.181701    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9cd8a2_5d2a_4b6e_a016_f771ca54da22.slice/crio-fe21d357d7a5428ef98a63d6e82560df5f2a281084a6cc96fbb03467134c8154 WatchSource:0}: Error finding container fe21d357d7a5428ef98a63d6e82560df5f2a281084a6cc96fbb03467134c8154: Status 404 returned error can't find the container with id fe21d357d7a5428ef98a63d6e82560df5f2a281084a6cc96fbb03467134c8154
Sep 11 18:23:30.697146 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:30.697110    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7851-88c088"]
Sep 11 18:23:30.700365 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:30.700229    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7851-88c088"]
Sep 11 18:23:31.101525 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:23:31.101490    2576 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7851-88c088" container="test-curl-7851-88c088"
Sep 11 18:23:31.102013 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:31.101690    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m" event={"ID":"ca9cd8a2-5d2a-4b6e-a016-f771ca54da22","Type":"ContainerStarted","Data":"fe21d357d7a5428ef98a63d6e82560df5f2a281084a6cc96fbb03467134c8154"}
Sep 11 18:23:31.472607 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:31.472527    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4"]
Sep 11 18:23:31.472995 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:31.472976    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c51afc09-c4e6-44be-9a3d-ddefddab2370" containerName="test-curl-7851-88c088"
Sep 11 18:23:31.473098 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:31.472998    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c51afc09-c4e6-44be-9a3d-ddefddab2370" containerName="test-curl-7851-88c088"
Sep 11 18:23:31.473156 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:31.473096    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c51afc09-c4e6-44be-9a3d-ddefddab2370" containerName="test-curl-7851-88c088"
Sep 11 18:23:31.480024 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:31.479995    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4"
Sep 11 18:23:31.482375 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:31.482352    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f63893-serving-cert\""
Sep 11 18:23:31.487673 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:31.486624    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4"]
Sep 11 18:23:31.573988 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:31.573944    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/092d7b84-cf77-4b84-b5cb-01b0b01a9a75-metrics-certs\") pod \"maas-api-e2e-inf-a-f63893-5545d46d88-442t4\" (UID: \"092d7b84-cf77-4b84-b5cb-01b0b01a9a75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4"
Sep 11 18:23:31.574188 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:31.574045    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/092d7b84-cf77-4b84-b5cb-01b0b01a9a75-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f63893-5545d46d88-442t4\" (UID: \"092d7b84-cf77-4b84-b5cb-01b0b01a9a75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4"
Sep 11 18:23:31.574188 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:31.574116    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/092d7b84-cf77-4b84-b5cb-01b0b01a9a75-maas-api-tls\") pod \"maas-api-e2e-inf-a-f63893-5545d46d88-442t4\" (UID: \"092d7b84-cf77-4b84-b5cb-01b0b01a9a75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4"
Sep 11 18:23:31.574188 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:31.574181    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfbnm\" (UniqueName: \"kubernetes.io/projected/092d7b84-cf77-4b84-b5cb-01b0b01a9a75-kube-api-access-sfbnm\") pod \"maas-api-e2e-inf-a-f63893-5545d46d88-442t4\" (UID: \"092d7b84-cf77-4b84-b5cb-01b0b01a9a75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4"
Sep 11 18:23:31.674935 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:31.674882    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/092d7b84-cf77-4b84-b5cb-01b0b01a9a75-metrics-certs\") pod \"maas-api-e2e-inf-a-f63893-5545d46d88-442t4\" (UID: \"092d7b84-cf77-4b84-b5cb-01b0b01a9a75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4"
Sep 11 18:23:31.675107 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:31.674945    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/092d7b84-cf77-4b84-b5cb-01b0b01a9a75-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f63893-5545d46d88-442t4\" (UID: \"092d7b84-cf77-4b84-b5cb-01b0b01a9a75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4"
Sep 11 18:23:31.675107 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:31.674979    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/092d7b84-cf77-4b84-b5cb-01b0b01a9a75-maas-api-tls\") pod \"maas-api-e2e-inf-a-f63893-5545d46d88-442t4\" (UID: \"092d7b84-cf77-4b84-b5cb-01b0b01a9a75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4"
Sep 11 18:23:31.675107 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:31.675055    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfbnm\" (UniqueName: \"kubernetes.io/projected/092d7b84-cf77-4b84-b5cb-01b0b01a9a75-kube-api-access-sfbnm\") pod \"maas-api-e2e-inf-a-f63893-5545d46d88-442t4\" (UID: \"092d7b84-cf77-4b84-b5cb-01b0b01a9a75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4"
Sep 11 18:23:31.675818 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:31.675789    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/092d7b84-cf77-4b84-b5cb-01b0b01a9a75-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f63893-5545d46d88-442t4\" (UID: \"092d7b84-cf77-4b84-b5cb-01b0b01a9a75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4"
Sep 11 18:23:31.678067 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:31.678040    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/092d7b84-cf77-4b84-b5cb-01b0b01a9a75-maas-api-tls\") pod \"maas-api-e2e-inf-a-f63893-5545d46d88-442t4\" (UID: \"092d7b84-cf77-4b84-b5cb-01b0b01a9a75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4"
Sep 11 18:23:31.678205 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:31.678132    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/092d7b84-cf77-4b84-b5cb-01b0b01a9a75-metrics-certs\") pod \"maas-api-e2e-inf-a-f63893-5545d46d88-442t4\" (UID: \"092d7b84-cf77-4b84-b5cb-01b0b01a9a75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4"
Sep 11 18:23:31.684160 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:31.684116    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfbnm\" (UniqueName: \"kubernetes.io/projected/092d7b84-cf77-4b84-b5cb-01b0b01a9a75-kube-api-access-sfbnm\") pod \"maas-api-e2e-inf-a-f63893-5545d46d88-442t4\" (UID: \"092d7b84-cf77-4b84-b5cb-01b0b01a9a75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4"
Sep 11 18:23:31.796515 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:31.796419    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4"
Sep 11 18:23:32.107615 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:32.107551    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2" event={"ID":"13435514-2ef4-401b-b2d6-cfe8fd0f52ef","Type":"ContainerStarted","Data":"c93fae0d29061d82eb009271eaa4ed6b9e2650af11e9d6bb72b935d3eb7622db"}
Sep 11 18:23:32.108017 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:32.107663    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7851-88c088" podUID="c51afc09-c4e6-44be-9a3d-ddefddab2370" containerName="test-curl-7851-88c088" containerID="cri-o://0c3efd3a3097de66e4dc7257492ce1920fa0139a3057505068780726356633aa" gracePeriod=2
Sep 11 18:23:32.124217 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:32.124135    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2" podStartSLOduration=1.635026124 podStartE2EDuration="3.124112737s" totalImagesPullingTime="1.489086613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:32.12340647 +0000 UTC m=+810.724767633" watchObservedRunningTime="2026-09-11 18:23:32.124112737 +0000 UTC m=+810.725473900"
Sep 11 18:23:32.140318 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:32.140272    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8"]
Sep 11 18:23:32.143632 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:32.143607    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8"
Sep 11 18:23:32.145921 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:32.145897    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f63893\""
Sep 11 18:23:32.146042 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:32.145926    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f63893-dockercfg-j8kvn\""
Sep 11 18:23:32.164429 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:32.164398    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8"]
Sep 11 18:23:32.184217 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:32.184186    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24fmp\" (UniqueName: \"kubernetes.io/projected/56b76440-2d65-4538-9cf9-72c2fdaf3f1c-kube-api-access-24fmp\") pod \"payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8\" (UID: \"56b76440-2d65-4538-9cf9-72c2fdaf3f1c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8"
Sep 11 18:23:32.184378 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:32.184286    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b76440-2d65-4538-9cf9-72c2fdaf3f1c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8\" (UID: \"56b76440-2d65-4538-9cf9-72c2fdaf3f1c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8"
Sep 11 18:23:32.285634 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:32.285593    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24fmp\" (UniqueName: \"kubernetes.io/projected/56b76440-2d65-4538-9cf9-72c2fdaf3f1c-kube-api-access-24fmp\") pod \"payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8\" (UID: \"56b76440-2d65-4538-9cf9-72c2fdaf3f1c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8"
Sep 11 18:23:32.285812 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:32.285658    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b76440-2d65-4538-9cf9-72c2fdaf3f1c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8\" (UID: \"56b76440-2d65-4538-9cf9-72c2fdaf3f1c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8"
Sep 11 18:23:32.286499 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:32.286467    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b76440-2d65-4538-9cf9-72c2fdaf3f1c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8\" (UID: \"56b76440-2d65-4538-9cf9-72c2fdaf3f1c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8"
Sep 11 18:23:32.293760 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:32.293724    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24fmp\" (UniqueName: \"kubernetes.io/projected/56b76440-2d65-4538-9cf9-72c2fdaf3f1c-kube-api-access-24fmp\") pod \"payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8\" (UID: \"56b76440-2d65-4538-9cf9-72c2fdaf3f1c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8"
Sep 11 18:23:32.460488 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:32.460441    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8"
Sep 11 18:23:32.706367 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:32.706341    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-f68ca2"
Sep 11 18:23:32.790150 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:32.790115    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8pvt\" (UniqueName: \"kubernetes.io/projected/1ea67197-a088-44d9-ab49-156a437ef12f-kube-api-access-q8pvt\") pod \"1ea67197-a088-44d9-ab49-156a437ef12f\" (UID: \"1ea67197-a088-44d9-ab49-156a437ef12f\") "
Sep 11 18:23:32.792365 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:32.792341    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea67197-a088-44d9-ab49-156a437ef12f-kube-api-access-q8pvt" (OuterVolumeSpecName: "kube-api-access-q8pvt") pod "1ea67197-a088-44d9-ab49-156a437ef12f" (UID: "1ea67197-a088-44d9-ab49-156a437ef12f"). InnerVolumeSpecName "kube-api-access-q8pvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:23:32.842564 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:32.842535    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8"]
Sep 11 18:23:32.845488 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:23:32.845448    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b76440_2d65_4538_9cf9_72c2fdaf3f1c.slice/crio-7d097fe687edc7f5452e978580814a00238c9cc47d7e2909b8f724b03dba92ca WatchSource:0}: Error finding container 7d097fe687edc7f5452e978580814a00238c9cc47d7e2909b8f724b03dba92ca: Status 404 returned error can't find the container with id 7d097fe687edc7f5452e978580814a00238c9cc47d7e2909b8f724b03dba92ca
Sep 11 18:23:32.845964 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:32.845940    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4"]
Sep 11 18:23:32.849105 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:23:32.849078    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092d7b84_cf77_4b84_b5cb_01b0b01a9a75.slice/crio-c9b6e3216a76e4bacee383fb3b4738008c81eb4d39c4ba3ad1a998b64c94640e WatchSource:0}: Error finding container c9b6e3216a76e4bacee383fb3b4738008c81eb4d39c4ba3ad1a998b64c94640e: Status 404 returned error can't find the container with id c9b6e3216a76e4bacee383fb3b4738008c81eb4d39c4ba3ad1a998b64c94640e
Sep 11 18:23:32.890756 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:32.890720    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8pvt\" (UniqueName: \"kubernetes.io/projected/1ea67197-a088-44d9-ab49-156a437ef12f-kube-api-access-q8pvt\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:23:33.115378 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:33.115279    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8" event={"ID":"56b76440-2d65-4538-9cf9-72c2fdaf3f1c","Type":"ContainerStarted","Data":"2705c6c9f9087120fb74b57397e36cd687201d5df5edfaa984b04eade516d4aa"}
Sep 11 18:23:33.115776 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:33.115430    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8"
Sep 11 18:23:33.115776 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:33.115477    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8" event={"ID":"56b76440-2d65-4538-9cf9-72c2fdaf3f1c","Type":"ContainerStarted","Data":"7d097fe687edc7f5452e978580814a00238c9cc47d7e2909b8f724b03dba92ca"}
Sep 11 18:23:33.117236 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:33.117209    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m" event={"ID":"ca9cd8a2-5d2a-4b6e-a016-f771ca54da22","Type":"ContainerStarted","Data":"ea34069dd3b25641ab249952ef3dd402be094ae14630e09b359ab9ec2ddc7bc5"}
Sep 11 18:23:33.117383 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:33.117267    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m"
Sep 11 18:23:33.118453 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:33.118429    2576 generic.go:356] "Generic (PLEG): container finished" podID="1ea67197-a088-44d9-ab49-156a437ef12f" containerID="e5e684ad2c1e0439e9f7fb80d4de9b4e164f5be33a946fcff2e97133ad409d38" exitCode=137
Sep 11 18:23:33.118567 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:33.118526    2576 scope.go:117] "RemoveContainer" containerID="e5e684ad2c1e0439e9f7fb80d4de9b4e164f5be33a946fcff2e97133ad409d38"
Sep 11 18:23:33.118567 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:33.118532    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-f68ca2"
Sep 11 18:23:33.120025 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:33.119999    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4" event={"ID":"092d7b84-cf77-4b84-b5cb-01b0b01a9a75","Type":"ContainerStarted","Data":"c9b6e3216a76e4bacee383fb3b4738008c81eb4d39c4ba3ad1a998b64c94640e"}
Sep 11 18:23:33.120145 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:33.120107    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2"
Sep 11 18:23:33.128251 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:33.128235    2576 scope.go:117] "RemoveContainer" containerID="e5e684ad2c1e0439e9f7fb80d4de9b4e164f5be33a946fcff2e97133ad409d38"
Sep 11 18:23:33.128533 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:23:33.128512    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5e684ad2c1e0439e9f7fb80d4de9b4e164f5be33a946fcff2e97133ad409d38\": container with ID starting with e5e684ad2c1e0439e9f7fb80d4de9b4e164f5be33a946fcff2e97133ad409d38 not found: ID does not exist" containerID="e5e684ad2c1e0439e9f7fb80d4de9b4e164f5be33a946fcff2e97133ad409d38"
Sep 11 18:23:33.128578 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:33.128541    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5e684ad2c1e0439e9f7fb80d4de9b4e164f5be33a946fcff2e97133ad409d38"} err="failed to get container status \"e5e684ad2c1e0439e9f7fb80d4de9b4e164f5be33a946fcff2e97133ad409d38\": rpc error: code = NotFound desc = could not find container \"e5e684ad2c1e0439e9f7fb80d4de9b4e164f5be33a946fcff2e97133ad409d38\": container with ID starting with e5e684ad2c1e0439e9f7fb80d4de9b4e164f5be33a946fcff2e97133ad409d38 not found: ID does not exist"
Sep 11 18:23:33.140488 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:23:33.140452    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7851-f68ca2\" is forbidden: User \"system:node:ip-10-0-134-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-111.ec2.internal' and this object" podUID="1ea67197-a088-44d9-ab49-156a437ef12f" pod="opendatahub/test-curl-7851-f68ca2"
Sep 11 18:23:33.141130 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:33.141085    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8" podStartSLOduration=1.1410715439999999 podStartE2EDuration="1.141071544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:33.13840089 +0000 UTC m=+811.739762053" watchObservedRunningTime="2026-09-11 18:23:33.141071544 +0000 UTC m=+811.742432707"
Sep 11 18:23:33.910539 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:33.910502    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ea67197-a088-44d9-ab49-156a437ef12f" path="/var/lib/kubelet/pods/1ea67197-a088-44d9-ab49-156a437ef12f/volumes"
Sep 11 18:23:34.127144 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:34.127111    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4" event={"ID":"092d7b84-cf77-4b84-b5cb-01b0b01a9a75","Type":"ContainerStarted","Data":"454ef1ee6eec3d11f43e020ba75041d6d1e205c8091f7ee6baa2c1e9a083f9cd"}
Sep 11 18:23:34.127558 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:34.127266    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4"
Sep 11 18:23:34.144924 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:34.144856    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m" podStartSLOduration=2.7134267149999998 podStartE2EDuration="5.144838218s" totalImagesPullingTime="2.431411503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:33.207498939 +0000 UTC m=+811.808860102" watchObservedRunningTime="2026-09-11 18:23:34.144838218 +0000 UTC m=+812.746199386"
Sep 11 18:23:34.145705 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:34.145676    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4" podStartSLOduration=2.8048162789999997 podStartE2EDuration="3.145667071s" totalImagesPullingTime="340.850792ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:34.143391494 +0000 UTC m=+812.744752658" watchObservedRunningTime="2026-09-11 18:23:34.145667071 +0000 UTC m=+812.747028234"
Sep 11 18:23:34.298708 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:34.298683    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-88c088"
Sep 11 18:23:34.405221 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:34.405137    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpx5z\" (UniqueName: \"kubernetes.io/projected/c51afc09-c4e6-44be-9a3d-ddefddab2370-kube-api-access-qpx5z\") pod \"c51afc09-c4e6-44be-9a3d-ddefddab2370\" (UID: \"c51afc09-c4e6-44be-9a3d-ddefddab2370\") "
Sep 11 18:23:34.407386 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:34.407355    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51afc09-c4e6-44be-9a3d-ddefddab2370-kube-api-access-qpx5z" (OuterVolumeSpecName: "kube-api-access-qpx5z") pod "c51afc09-c4e6-44be-9a3d-ddefddab2370" (UID: "c51afc09-c4e6-44be-9a3d-ddefddab2370"). InnerVolumeSpecName "kube-api-access-qpx5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:23:34.506264 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:34.506228    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpx5z\" (UniqueName: \"kubernetes.io/projected/c51afc09-c4e6-44be-9a3d-ddefddab2370-kube-api-access-qpx5z\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:23:35.133003 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:35.132966    2576 generic.go:356] "Generic (PLEG): container finished" podID="c51afc09-c4e6-44be-9a3d-ddefddab2370" containerID="0c3efd3a3097de66e4dc7257492ce1920fa0139a3057505068780726356633aa" exitCode=137
Sep 11 18:23:35.133411 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:35.133010    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7851-88c088"
Sep 11 18:23:35.133411 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:35.133064    2576 scope.go:117] "RemoveContainer" containerID="0c3efd3a3097de66e4dc7257492ce1920fa0139a3057505068780726356633aa"
Sep 11 18:23:35.142273 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:35.142256    2576 scope.go:117] "RemoveContainer" containerID="0c3efd3a3097de66e4dc7257492ce1920fa0139a3057505068780726356633aa"
Sep 11 18:23:35.142525 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:23:35.142507    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c3efd3a3097de66e4dc7257492ce1920fa0139a3057505068780726356633aa\": container with ID starting with 0c3efd3a3097de66e4dc7257492ce1920fa0139a3057505068780726356633aa not found: ID does not exist" containerID="0c3efd3a3097de66e4dc7257492ce1920fa0139a3057505068780726356633aa"
Sep 11 18:23:35.142573 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:35.142533    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c3efd3a3097de66e4dc7257492ce1920fa0139a3057505068780726356633aa"} err="failed to get container status \"0c3efd3a3097de66e4dc7257492ce1920fa0139a3057505068780726356633aa\": rpc error: code = NotFound desc = could not find container \"0c3efd3a3097de66e4dc7257492ce1920fa0139a3057505068780726356633aa\": container with ID starting with 0c3efd3a3097de66e4dc7257492ce1920fa0139a3057505068780726356633aa not found: ID does not exist"
Sep 11 18:23:35.144199 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:23:35.144171    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7851-88c088\" is forbidden: User \"system:node:ip-10-0-134-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-111.ec2.internal' and this object" podUID="c51afc09-c4e6-44be-9a3d-ddefddab2370" pod="opendatahub/test-curl-7851-88c088"
Sep 11 18:23:35.910843 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:35.910809    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c51afc09-c4e6-44be-9a3d-ddefddab2370" path="/var/lib/kubelet/pods/c51afc09-c4e6-44be-9a3d-ddefddab2370/volumes"
Sep 11 18:23:39.134226 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:39.134196    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m"
Sep 11 18:23:39.256856 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:39.256818    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw"]
Sep 11 18:23:39.261373 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:39.261355    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw"
Sep 11 18:23:39.263618 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:39.263572    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2f7a9131\""
Sep 11 18:23:39.264043 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:39.264024    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2f7a9131-dockercfg-6lc8l\""
Sep 11 18:23:39.269696 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:39.269673    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw"]
Sep 11 18:23:39.345900 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:39.345861    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz9cc\" (UniqueName: \"kubernetes.io/projected/92b3ceb2-2781-4c1a-bcee-a305ff196fb0-kube-api-access-kz9cc\") pod \"payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw\" (UID: \"92b3ceb2-2781-4c1a-bcee-a305ff196fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw"
Sep 11 18:23:39.346075 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:39.345985    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92b3ceb2-2781-4c1a-bcee-a305ff196fb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw\" (UID: \"92b3ceb2-2781-4c1a-bcee-a305ff196fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw"
Sep 11 18:23:39.446489 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:39.446444    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92b3ceb2-2781-4c1a-bcee-a305ff196fb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw\" (UID: \"92b3ceb2-2781-4c1a-bcee-a305ff196fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw"
Sep 11 18:23:39.447530 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:39.447501    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92b3ceb2-2781-4c1a-bcee-a305ff196fb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw\" (UID: \"92b3ceb2-2781-4c1a-bcee-a305ff196fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw"
Sep 11 18:23:39.447664 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:39.447621    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz9cc\" (UniqueName: \"kubernetes.io/projected/92b3ceb2-2781-4c1a-bcee-a305ff196fb0-kube-api-access-kz9cc\") pod \"payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw\" (UID: \"92b3ceb2-2781-4c1a-bcee-a305ff196fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw"
Sep 11 18:23:39.455610 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:39.455574    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz9cc\" (UniqueName: \"kubernetes.io/projected/92b3ceb2-2781-4c1a-bcee-a305ff196fb0-kube-api-access-kz9cc\") pod \"payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw\" (UID: \"92b3ceb2-2781-4c1a-bcee-a305ff196fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw"
Sep 11 18:23:39.573297 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:39.573259    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw"
Sep 11 18:23:39.843017 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:39.842847    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw"]
Sep 11 18:23:39.845648 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:23:39.845613    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b3ceb2_2781_4c1a_bcee_a305ff196fb0.slice/crio-6f4b540f0dff1916d7a3b0d77850722a30331f54d329d78c26698ad9f4d27956 WatchSource:0}: Error finding container 6f4b540f0dff1916d7a3b0d77850722a30331f54d329d78c26698ad9f4d27956: Status 404 returned error can't find the container with id 6f4b540f0dff1916d7a3b0d77850722a30331f54d329d78c26698ad9f4d27956
Sep 11 18:23:40.139104 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:40.139025    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4"
Sep 11 18:23:40.157136 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:40.157102    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw" event={"ID":"92b3ceb2-2781-4c1a-bcee-a305ff196fb0","Type":"ContainerStarted","Data":"cd5552f7c38aa7884bee7ede41f6e34c9e5938ad360021c844d71a644ff847bf"}
Sep 11 18:23:40.157313 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:40.157159    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw" event={"ID":"92b3ceb2-2781-4c1a-bcee-a305ff196fb0","Type":"ContainerStarted","Data":"6f4b540f0dff1916d7a3b0d77850722a30331f54d329d78c26698ad9f4d27956"}
Sep 11 18:23:40.157313 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:40.157182    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw"
Sep 11 18:23:40.182637 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:40.182557    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw" podStartSLOduration=1.182534545 podStartE2EDuration="1.182534545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:40.180934622 +0000 UTC m=+818.782295795" watchObservedRunningTime="2026-09-11 18:23:40.182534545 +0000 UTC m=+818.783895709"
Sep 11 18:23:41.327036 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:41.327001    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc"]
Sep 11 18:23:41.330789 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:41.330760    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc"
Sep 11 18:23:41.333327 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:41.333304    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-89bb6c\""
Sep 11 18:23:41.333427 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:41.333347    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-89bb6c-dockercfg-6jprb\""
Sep 11 18:23:41.342821 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:41.342795    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc"]
Sep 11 18:23:41.364127 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:41.364085    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb2a3574-ea6e-4b22-9aa8-f103e36a47c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc\" (UID: \"cb2a3574-ea6e-4b22-9aa8-f103e36a47c0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc"
Sep 11 18:23:41.364294 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:41.364144    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5ks8\" (UniqueName: \"kubernetes.io/projected/cb2a3574-ea6e-4b22-9aa8-f103e36a47c0-kube-api-access-m5ks8\") pod \"payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc\" (UID: \"cb2a3574-ea6e-4b22-9aa8-f103e36a47c0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc"
Sep 11 18:23:41.368400 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:41.368373    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb"]
Sep 11 18:23:41.371738 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:41.371717    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb"
Sep 11 18:23:41.382978 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:41.382954    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb"]
Sep 11 18:23:41.464785 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:41.464749    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6wp6\" (UniqueName: \"kubernetes.io/projected/d5ba777e-d6e1-4b50-a97c-3155a80a2400-kube-api-access-l6wp6\") pod \"payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb\" (UID: \"d5ba777e-d6e1-4b50-a97c-3155a80a2400\") " pod="openshift-ingress/payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb"
Sep 11 18:23:41.464785 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:41.464795    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb2a3574-ea6e-4b22-9aa8-f103e36a47c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc\" (UID: \"cb2a3574-ea6e-4b22-9aa8-f103e36a47c0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc"
Sep 11 18:23:41.465024 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:41.464814    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ba777e-d6e1-4b50-a97c-3155a80a2400-plugins-config-volume\") pod \"payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb\" (UID: \"d5ba777e-d6e1-4b50-a97c-3155a80a2400\") " pod="openshift-ingress/payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb"
Sep 11 18:23:41.465024 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:41.464844    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5ks8\" (UniqueName: \"kubernetes.io/projected/cb2a3574-ea6e-4b22-9aa8-f103e36a47c0-kube-api-access-m5ks8\") pod \"payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc\" (UID: \"cb2a3574-ea6e-4b22-9aa8-f103e36a47c0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc"
Sep 11 18:23:41.465485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:41.465465    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb2a3574-ea6e-4b22-9aa8-f103e36a47c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc\" (UID: \"cb2a3574-ea6e-4b22-9aa8-f103e36a47c0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc"
Sep 11 18:23:41.473416 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:41.473386    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5ks8\" (UniqueName: \"kubernetes.io/projected/cb2a3574-ea6e-4b22-9aa8-f103e36a47c0-kube-api-access-m5ks8\") pod \"payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc\" (UID: \"cb2a3574-ea6e-4b22-9aa8-f103e36a47c0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc"
Sep 11 18:23:41.565291 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:41.565258    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6wp6\" (UniqueName: \"kubernetes.io/projected/d5ba777e-d6e1-4b50-a97c-3155a80a2400-kube-api-access-l6wp6\") pod \"payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb\" (UID: \"d5ba777e-d6e1-4b50-a97c-3155a80a2400\") " pod="openshift-ingress/payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb"
Sep 11 18:23:41.565291 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:41.565296    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ba777e-d6e1-4b50-a97c-3155a80a2400-plugins-config-volume\") pod \"payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb\" (UID: \"d5ba777e-d6e1-4b50-a97c-3155a80a2400\") " pod="openshift-ingress/payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb"
Sep 11 18:23:41.566030 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:41.566012    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ba777e-d6e1-4b50-a97c-3155a80a2400-plugins-config-volume\") pod \"payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb\" (UID: \"d5ba777e-d6e1-4b50-a97c-3155a80a2400\") " pod="openshift-ingress/payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb"
Sep 11 18:23:41.573539 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:41.573511    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6wp6\" (UniqueName: \"kubernetes.io/projected/d5ba777e-d6e1-4b50-a97c-3155a80a2400-kube-api-access-l6wp6\") pod \"payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb\" (UID: \"d5ba777e-d6e1-4b50-a97c-3155a80a2400\") " pod="openshift-ingress/payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb"
Sep 11 18:23:41.643346 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:41.643262    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc"
Sep 11 18:23:41.682752 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:41.682719    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb"
Sep 11 18:23:41.916792 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:41.916716    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc"]
Sep 11 18:23:41.919395 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:23:41.919369    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2a3574_ea6e_4b22_9aa8_f103e36a47c0.slice/crio-8a649c151b0cc60c05414b5f85fa070977ca6f1f86311f987b4c63ef903103ea WatchSource:0}: Error finding container 8a649c151b0cc60c05414b5f85fa070977ca6f1f86311f987b4c63ef903103ea: Status 404 returned error can't find the container with id 8a649c151b0cc60c05414b5f85fa070977ca6f1f86311f987b4c63ef903103ea
Sep 11 18:23:41.962396 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:41.962274    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb"]
Sep 11 18:23:41.964553 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:23:41.964523    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ba777e_d6e1_4b50_a97c_3155a80a2400.slice/crio-a8b574a62aa92eef0260209b400a9d84936d86791f74a7e1cc4668d44038702e WatchSource:0}: Error finding container a8b574a62aa92eef0260209b400a9d84936d86791f74a7e1cc4668d44038702e: Status 404 returned error can't find the container with id a8b574a62aa92eef0260209b400a9d84936d86791f74a7e1cc4668d44038702e
Sep 11 18:23:42.172724 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:42.172624    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc" event={"ID":"cb2a3574-ea6e-4b22-9aa8-f103e36a47c0","Type":"ContainerStarted","Data":"1f6eb0fdbaf18e327f3e34cff226d415d22b3442865f60d98dd007d7ea910a21"}
Sep 11 18:23:42.172724 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:42.172695    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc" event={"ID":"cb2a3574-ea6e-4b22-9aa8-f103e36a47c0","Type":"ContainerStarted","Data":"8a649c151b0cc60c05414b5f85fa070977ca6f1f86311f987b4c63ef903103ea"}
Sep 11 18:23:42.172953 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:42.172753    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc"
Sep 11 18:23:42.174637 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:42.174605    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb" event={"ID":"d5ba777e-d6e1-4b50-a97c-3155a80a2400","Type":"ContainerStarted","Data":"9477f0feeec226cc6af56dec1d15a2b3bfa1c8530e6b0689fb8550ea4bcb67e6"}
Sep 11 18:23:42.174771 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:42.174656    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb" event={"ID":"d5ba777e-d6e1-4b50-a97c-3155a80a2400","Type":"ContainerStarted","Data":"a8b574a62aa92eef0260209b400a9d84936d86791f74a7e1cc4668d44038702e"}
Sep 11 18:23:42.174771 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:42.174684    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb"
Sep 11 18:23:42.193129 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:42.193081    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc" podStartSLOduration=1.193064569 podStartE2EDuration="1.193064569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:42.192473673 +0000 UTC m=+820.793834837" watchObservedRunningTime="2026-09-11 18:23:42.193064569 +0000 UTC m=+820.794425732"
Sep 11 18:23:42.211990 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:42.211933    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb" podStartSLOduration=1.211914583 podStartE2EDuration="1.211914583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:42.20987341 +0000 UTC m=+820.811234575" watchObservedRunningTime="2026-09-11 18:23:42.211914583 +0000 UTC m=+820.813275756"
Sep 11 18:23:44.129740 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:44.129707    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8"
Sep 11 18:23:44.130104 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:44.129789    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2"
Sep 11 18:23:48.434822 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:48.434787    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:23:51.166747 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:51.166717    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw"
Sep 11 18:23:51.685806 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:51.685771    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"]
Sep 11 18:23:51.691683 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:51.691659    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"
Sep 11 18:23:51.693990 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:51.693966    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-12eb53-serving-cert\""
Sep 11 18:23:51.698900 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:51.698875    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"]
Sep 11 18:23:51.757994 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:51.757949    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-maas-api-tls\") pod \"maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6\" (UID: \"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"
Sep 11 18:23:51.757994 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:51.757993    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-metrics-certs\") pod \"maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6\" (UID: \"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"
Sep 11 18:23:51.758244 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:51.758059    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbgdg\" (UniqueName: \"kubernetes.io/projected/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-kube-api-access-bbgdg\") pod \"maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6\" (UID: \"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"
Sep 11 18:23:51.758244 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:51.758084    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6\" (UID: \"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"
Sep 11 18:23:51.858917 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:51.858882    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-maas-api-tls\") pod \"maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6\" (UID: \"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"
Sep 11 18:23:51.858917 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:51.858919    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-metrics-certs\") pod \"maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6\" (UID: \"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"
Sep 11 18:23:51.859149 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:51.858940    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbgdg\" (UniqueName: \"kubernetes.io/projected/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-kube-api-access-bbgdg\") pod \"maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6\" (UID: \"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"
Sep 11 18:23:51.859149 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:51.858956    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6\" (UID: \"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"
Sep 11 18:23:51.859149 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:23:51.859094    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-serving-cert: secret "maas-api-e2e-inf-b-12eb53-serving-cert" not found
Sep 11 18:23:51.859267 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:23:51.859172    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-maas-api-tls podName:8bf1b9de-9a29-4fbf-9ace-ea260daabbf8 nodeName:}" failed. No retries permitted until 2026-09-11 18:23:52.359150107 +0000 UTC m=+830.960511265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-maas-api-tls") pod "maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6" (UID: "8bf1b9de-9a29-4fbf-9ace-ea260daabbf8") : secret "maas-api-e2e-inf-b-12eb53-serving-cert" not found
Sep 11 18:23:51.859680 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:51.859659    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6\" (UID: \"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"
Sep 11 18:23:51.861629 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:51.861606    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-metrics-certs\") pod \"maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6\" (UID: \"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"
Sep 11 18:23:51.875061 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:51.875033    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbgdg\" (UniqueName: \"kubernetes.io/projected/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-kube-api-access-bbgdg\") pod \"maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6\" (UID: \"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"
Sep 11 18:23:52.363185 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:52.363138    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-maas-api-tls\") pod \"maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6\" (UID: \"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"
Sep 11 18:23:52.365761 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:52.365731    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-maas-api-tls\") pod \"maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6\" (UID: \"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"
Sep 11 18:23:52.484211 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:52.484175    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2"]
Sep 11 18:23:52.488770 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:52.488749    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2"
Sep 11 18:23:52.491245 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:52.491220    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-12eb53\""
Sep 11 18:23:52.507349 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:52.507246    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2"]
Sep 11 18:23:52.565627 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:52.565560    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qfh2\" (UniqueName: \"kubernetes.io/projected/3ac93d8a-158d-42b3-8b85-3217f8fab901-kube-api-access-7qfh2\") pod \"payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2\" (UID: \"3ac93d8a-158d-42b3-8b85-3217f8fab901\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2"
Sep 11 18:23:52.565817 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:52.565650    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac93d8a-158d-42b3-8b85-3217f8fab901-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2\" (UID: \"3ac93d8a-158d-42b3-8b85-3217f8fab901\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2"
Sep 11 18:23:52.603981 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:52.603946    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"
Sep 11 18:23:52.667400 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:52.667338    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qfh2\" (UniqueName: \"kubernetes.io/projected/3ac93d8a-158d-42b3-8b85-3217f8fab901-kube-api-access-7qfh2\") pod \"payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2\" (UID: \"3ac93d8a-158d-42b3-8b85-3217f8fab901\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2"
Sep 11 18:23:52.667570 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:52.667405    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac93d8a-158d-42b3-8b85-3217f8fab901-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2\" (UID: \"3ac93d8a-158d-42b3-8b85-3217f8fab901\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2"
Sep 11 18:23:52.668477 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:52.668448    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac93d8a-158d-42b3-8b85-3217f8fab901-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2\" (UID: \"3ac93d8a-158d-42b3-8b85-3217f8fab901\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2"
Sep 11 18:23:52.675937 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:52.675914    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qfh2\" (UniqueName: \"kubernetes.io/projected/3ac93d8a-158d-42b3-8b85-3217f8fab901-kube-api-access-7qfh2\") pod \"payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2\" (UID: \"3ac93d8a-158d-42b3-8b85-3217f8fab901\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2"
Sep 11 18:23:52.800296 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:52.800249    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2"
Sep 11 18:23:52.877323 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:52.877291    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"]
Sep 11 18:23:52.879390 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:23:52.879364    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf1b9de_9a29_4fbf_9ace_ea260daabbf8.slice/crio-3111fa85ef4898062e98d7826a0373af7dd25610a2a0b6f2d940621362179b4e WatchSource:0}: Error finding container 3111fa85ef4898062e98d7826a0373af7dd25610a2a0b6f2d940621362179b4e: Status 404 returned error can't find the container with id 3111fa85ef4898062e98d7826a0373af7dd25610a2a0b6f2d940621362179b4e
Sep 11 18:23:53.068187 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:53.068158    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2"]
Sep 11 18:23:53.071045 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:23:53.071021    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac93d8a_158d_42b3_8b85_3217f8fab901.slice/crio-c03ab51f42d853c56412e6dfb66f5f73e6e5af432a8c3376447aae7437f3b314 WatchSource:0}: Error finding container c03ab51f42d853c56412e6dfb66f5f73e6e5af432a8c3376447aae7437f3b314: Status 404 returned error can't find the container with id c03ab51f42d853c56412e6dfb66f5f73e6e5af432a8c3376447aae7437f3b314
Sep 11 18:23:53.181963 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:53.181939    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc"
Sep 11 18:23:53.182088 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:53.182009    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb"
Sep 11 18:23:53.238058 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:53.236842    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6" event={"ID":"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8","Type":"ContainerStarted","Data":"3111fa85ef4898062e98d7826a0373af7dd25610a2a0b6f2d940621362179b4e"}
Sep 11 18:23:53.239800 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:53.239334    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2" event={"ID":"3ac93d8a-158d-42b3-8b85-3217f8fab901","Type":"ContainerStarted","Data":"27e1c4d5299122e1bb748fd5a73f1f896a349e9c6a345c59dfe7f4cd5a24824d"}
Sep 11 18:23:53.239800 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:53.239388    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2" event={"ID":"3ac93d8a-158d-42b3-8b85-3217f8fab901","Type":"ContainerStarted","Data":"c03ab51f42d853c56412e6dfb66f5f73e6e5af432a8c3376447aae7437f3b314"}
Sep 11 18:23:53.239800 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:53.239733    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2"
Sep 11 18:23:53.257619 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:53.257558    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2" podStartSLOduration=1.2575443069999999 podStartE2EDuration="1.257544307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:53.256014645 +0000 UTC m=+831.857375803" watchObservedRunningTime="2026-09-11 18:23:53.257544307 +0000 UTC m=+831.858905469"
Sep 11 18:23:54.245976 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:54.245943    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6" event={"ID":"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8","Type":"ContainerStarted","Data":"5f57224806651c5e095de3d1ac54341ad707adc98d581a9de30cc289b0d847d4"}
Sep 11 18:23:54.246390 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:54.246201    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"
Sep 11 18:23:54.264983 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:23:54.264929    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6" podStartSLOduration=2.9635469309999998 podStartE2EDuration="3.264913759s" totalImagesPullingTime="301.366828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:54.262995442 +0000 UTC m=+832.864356632" watchObservedRunningTime="2026-09-11 18:23:54.264913759 +0000 UTC m=+832.866274922"
Sep 11 18:24:00.256257 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:00.256225    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"
Sep 11 18:24:04.248358 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:04.248323    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2"
Sep 11 18:24:06.713161 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:06.713125    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw"]
Sep 11 18:24:06.713646 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:06.713383    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw" podUID="92b3ceb2-2781-4c1a-bcee-a305ff196fb0" containerName="payload-pre-processing" containerID="cri-o://cd5552f7c38aa7884bee7ede41f6e34c9e5938ad360021c844d71a644ff847bf" gracePeriod=30
Sep 11 18:24:06.958653 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:06.958623    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw"
Sep 11 18:24:07.093763 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:07.093671    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92b3ceb2-2781-4c1a-bcee-a305ff196fb0-plugins-config-volume\") pod \"92b3ceb2-2781-4c1a-bcee-a305ff196fb0\" (UID: \"92b3ceb2-2781-4c1a-bcee-a305ff196fb0\") "
Sep 11 18:24:07.093932 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:07.093785    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kz9cc\" (UniqueName: \"kubernetes.io/projected/92b3ceb2-2781-4c1a-bcee-a305ff196fb0-kube-api-access-kz9cc\") pod \"92b3ceb2-2781-4c1a-bcee-a305ff196fb0\" (UID: \"92b3ceb2-2781-4c1a-bcee-a305ff196fb0\") "
Sep 11 18:24:07.094136 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:07.094109    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b3ceb2-2781-4c1a-bcee-a305ff196fb0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "92b3ceb2-2781-4c1a-bcee-a305ff196fb0" (UID: "92b3ceb2-2781-4c1a-bcee-a305ff196fb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:24:07.095906 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:07.095885    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b3ceb2-2781-4c1a-bcee-a305ff196fb0-kube-api-access-kz9cc" (OuterVolumeSpecName: "kube-api-access-kz9cc") pod "92b3ceb2-2781-4c1a-bcee-a305ff196fb0" (UID: "92b3ceb2-2781-4c1a-bcee-a305ff196fb0"). InnerVolumeSpecName "kube-api-access-kz9cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:24:07.194497 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:07.194455    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz9cc\" (UniqueName: \"kubernetes.io/projected/92b3ceb2-2781-4c1a-bcee-a305ff196fb0-kube-api-access-kz9cc\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:24:07.194497 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:07.194492    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92b3ceb2-2781-4c1a-bcee-a305ff196fb0-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:24:07.318348 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:07.318311    2576 generic.go:356] "Generic (PLEG): container finished" podID="92b3ceb2-2781-4c1a-bcee-a305ff196fb0" containerID="cd5552f7c38aa7884bee7ede41f6e34c9e5938ad360021c844d71a644ff847bf" exitCode=0
Sep 11 18:24:07.318539 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:07.318375    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw"
Sep 11 18:24:07.318539 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:07.318396    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw" event={"ID":"92b3ceb2-2781-4c1a-bcee-a305ff196fb0","Type":"ContainerDied","Data":"cd5552f7c38aa7884bee7ede41f6e34c9e5938ad360021c844d71a644ff847bf"}
Sep 11 18:24:07.318539 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:07.318473    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw" event={"ID":"92b3ceb2-2781-4c1a-bcee-a305ff196fb0","Type":"ContainerDied","Data":"6f4b540f0dff1916d7a3b0d77850722a30331f54d329d78c26698ad9f4d27956"}
Sep 11 18:24:07.318539 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:07.318497    2576 scope.go:117] "RemoveContainer" containerID="cd5552f7c38aa7884bee7ede41f6e34c9e5938ad360021c844d71a644ff847bf"
Sep 11 18:24:07.327358 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:07.327341    2576 scope.go:117] "RemoveContainer" containerID="cd5552f7c38aa7884bee7ede41f6e34c9e5938ad360021c844d71a644ff847bf"
Sep 11 18:24:07.327620 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:24:07.327602    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd5552f7c38aa7884bee7ede41f6e34c9e5938ad360021c844d71a644ff847bf\": container with ID starting with cd5552f7c38aa7884bee7ede41f6e34c9e5938ad360021c844d71a644ff847bf not found: ID does not exist" containerID="cd5552f7c38aa7884bee7ede41f6e34c9e5938ad360021c844d71a644ff847bf"
Sep 11 18:24:07.327669 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:07.327627    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd5552f7c38aa7884bee7ede41f6e34c9e5938ad360021c844d71a644ff847bf"} err="failed to get container status \"cd5552f7c38aa7884bee7ede41f6e34c9e5938ad360021c844d71a644ff847bf\": rpc error: code = NotFound desc = could not find container \"cd5552f7c38aa7884bee7ede41f6e34c9e5938ad360021c844d71a644ff847bf\": container with ID starting with cd5552f7c38aa7884bee7ede41f6e34c9e5938ad360021c844d71a644ff847bf not found: ID does not exist"
Sep 11 18:24:07.339607 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:07.339566    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw"]
Sep 11 18:24:07.343731 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:07.343707    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2f7a9131-7964d9d4f8-rlrbw"]
Sep 11 18:24:07.910136 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:07.910106    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92b3ceb2-2781-4c1a-bcee-a305ff196fb0" path="/var/lib/kubelet/pods/92b3ceb2-2781-4c1a-bcee-a305ff196fb0/volumes"
Sep 11 18:24:14.328271 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:14.328236    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:24:14.821275 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:14.821241    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p"]
Sep 11 18:24:14.821617 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:14.821605    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92b3ceb2-2781-4c1a-bcee-a305ff196fb0" containerName="payload-pre-processing"
Sep 11 18:24:14.821665 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:14.821618    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="92b3ceb2-2781-4c1a-bcee-a305ff196fb0" containerName="payload-pre-processing"
Sep 11 18:24:14.821699 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:14.821672    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="92b3ceb2-2781-4c1a-bcee-a305ff196fb0" containerName="payload-pre-processing"
Sep 11 18:24:14.825843 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:14.825822    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p"
Sep 11 18:24:14.828597 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:14.828557    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-f8c01007\""
Sep 11 18:24:14.828710 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:14.828637    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-f8c01007-dockercfg-4gg59\""
Sep 11 18:24:14.836191 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:14.836167    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p"]
Sep 11 18:24:14.853260 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:14.853223    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9"]
Sep 11 18:24:14.856569 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:14.856541    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9"
Sep 11 18:24:14.870283 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:14.870257    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9"]
Sep 11 18:24:14.964916 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:14.964876    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0558cdf-cfd6-4f72-b450-1a4a29b1f87b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9\" (UID: \"d0558cdf-cfd6-4f72-b450-1a4a29b1f87b\") " pod="openshift-ingress/payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9"
Sep 11 18:24:14.964916 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:14.964915    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6rwz\" (UniqueName: \"kubernetes.io/projected/d0558cdf-cfd6-4f72-b450-1a4a29b1f87b-kube-api-access-q6rwz\") pod \"payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9\" (UID: \"d0558cdf-cfd6-4f72-b450-1a4a29b1f87b\") " pod="openshift-ingress/payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9"
Sep 11 18:24:14.965142 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:14.964948    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv88g\" (UniqueName: \"kubernetes.io/projected/2b2cd515-f0cd-449b-a5f4-4eea1c9ece71-kube-api-access-xv88g\") pod \"payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p\" (UID: \"2b2cd515-f0cd-449b-a5f4-4eea1c9ece71\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p"
Sep 11 18:24:14.965142 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:14.965044    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b2cd515-f0cd-449b-a5f4-4eea1c9ece71-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p\" (UID: \"2b2cd515-f0cd-449b-a5f4-4eea1c9ece71\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p"
Sep 11 18:24:15.066007 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:15.065962    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv88g\" (UniqueName: \"kubernetes.io/projected/2b2cd515-f0cd-449b-a5f4-4eea1c9ece71-kube-api-access-xv88g\") pod \"payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p\" (UID: \"2b2cd515-f0cd-449b-a5f4-4eea1c9ece71\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p"
Sep 11 18:24:15.066205 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:15.066049    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b2cd515-f0cd-449b-a5f4-4eea1c9ece71-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p\" (UID: \"2b2cd515-f0cd-449b-a5f4-4eea1c9ece71\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p"
Sep 11 18:24:15.066205 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:15.066120    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0558cdf-cfd6-4f72-b450-1a4a29b1f87b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9\" (UID: \"d0558cdf-cfd6-4f72-b450-1a4a29b1f87b\") " pod="openshift-ingress/payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9"
Sep 11 18:24:15.066205 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:15.066147    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6rwz\" (UniqueName: \"kubernetes.io/projected/d0558cdf-cfd6-4f72-b450-1a4a29b1f87b-kube-api-access-q6rwz\") pod \"payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9\" (UID: \"d0558cdf-cfd6-4f72-b450-1a4a29b1f87b\") " pod="openshift-ingress/payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9"
Sep 11 18:24:15.066812 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:15.066788    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b2cd515-f0cd-449b-a5f4-4eea1c9ece71-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p\" (UID: \"2b2cd515-f0cd-449b-a5f4-4eea1c9ece71\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p"
Sep 11 18:24:15.066880 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:15.066861    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0558cdf-cfd6-4f72-b450-1a4a29b1f87b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9\" (UID: \"d0558cdf-cfd6-4f72-b450-1a4a29b1f87b\") " pod="openshift-ingress/payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9"
Sep 11 18:24:15.074158 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:15.074094    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv88g\" (UniqueName: \"kubernetes.io/projected/2b2cd515-f0cd-449b-a5f4-4eea1c9ece71-kube-api-access-xv88g\") pod \"payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p\" (UID: \"2b2cd515-f0cd-449b-a5f4-4eea1c9ece71\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p"
Sep 11 18:24:15.074319 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:15.074301    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6rwz\" (UniqueName: \"kubernetes.io/projected/d0558cdf-cfd6-4f72-b450-1a4a29b1f87b-kube-api-access-q6rwz\") pod \"payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9\" (UID: \"d0558cdf-cfd6-4f72-b450-1a4a29b1f87b\") " pod="openshift-ingress/payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9"
Sep 11 18:24:15.136546 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:15.136510    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p"
Sep 11 18:24:15.165916 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:15.165878    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9"
Sep 11 18:24:15.412490 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:15.412464    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p"]
Sep 11 18:24:15.415288 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:24:15.415263    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2cd515_f0cd_449b_a5f4_4eea1c9ece71.slice/crio-138a732b740213e05f3b2fcc01bad5b78519f72dd01230f660e17ee59091a400 WatchSource:0}: Error finding container 138a732b740213e05f3b2fcc01bad5b78519f72dd01230f660e17ee59091a400: Status 404 returned error can't find the container with id 138a732b740213e05f3b2fcc01bad5b78519f72dd01230f660e17ee59091a400
Sep 11 18:24:15.449772 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:15.449734    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9"]
Sep 11 18:24:15.451470 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:24:15.451445    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0558cdf_cfd6_4f72_b450_1a4a29b1f87b.slice/crio-cefec8b5ff78592ad136d457e9bba08d5c8c5e1fdf8c2a5fe49e6d680fb366a4 WatchSource:0}: Error finding container cefec8b5ff78592ad136d457e9bba08d5c8c5e1fdf8c2a5fe49e6d680fb366a4: Status 404 returned error can't find the container with id cefec8b5ff78592ad136d457e9bba08d5c8c5e1fdf8c2a5fe49e6d680fb366a4
Sep 11 18:24:16.363568 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:16.363526    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9" event={"ID":"d0558cdf-cfd6-4f72-b450-1a4a29b1f87b","Type":"ContainerStarted","Data":"12a4283bc3a1c78cf591dff44fbea1af95baec4fc3eee5f327edb62fabc91ae3"}
Sep 11 18:24:16.363798 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:16.363618    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9" event={"ID":"d0558cdf-cfd6-4f72-b450-1a4a29b1f87b","Type":"ContainerStarted","Data":"cefec8b5ff78592ad136d457e9bba08d5c8c5e1fdf8c2a5fe49e6d680fb366a4"}
Sep 11 18:24:16.363798 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:16.363649    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9"
Sep 11 18:24:16.365316 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:16.365291    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p" event={"ID":"2b2cd515-f0cd-449b-a5f4-4eea1c9ece71","Type":"ContainerStarted","Data":"6e1ff1303cdbaa5b5e4edf7bbdf36ddb69e988e6848e9df2029ad7d873df1ebc"}
Sep 11 18:24:16.365421 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:16.365334    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p" event={"ID":"2b2cd515-f0cd-449b-a5f4-4eea1c9ece71","Type":"ContainerStarted","Data":"138a732b740213e05f3b2fcc01bad5b78519f72dd01230f660e17ee59091a400"}
Sep 11 18:24:16.365421 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:16.365357    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p"
Sep 11 18:24:16.385146 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:16.385094    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9" podStartSLOduration=2.385079182 podStartE2EDuration="2.385079182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:24:16.383878221 +0000 UTC m=+854.985239388" watchObservedRunningTime="2026-09-11 18:24:16.385079182 +0000 UTC m=+854.986440346"
Sep 11 18:24:16.401461 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:16.401403    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p" podStartSLOduration=2.401389956 podStartE2EDuration="2.401389956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:24:16.400276072 +0000 UTC m=+855.001637239" watchObservedRunningTime="2026-09-11 18:24:16.401389956 +0000 UTC m=+855.002751119"
Sep 11 18:24:27.372647 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:27.372615    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9"
Sep 11 18:24:27.373097 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:27.372800    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p"
Sep 11 18:24:27.511511 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:27.511472    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:24:39.831111 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:39.831075    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:24:52.536957 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:52.536923    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9"]
Sep 11 18:24:52.537389 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:52.537169    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9" podUID="d0558cdf-cfd6-4f72-b450-1a4a29b1f87b" containerName="payload-processing" containerID="cri-o://12a4283bc3a1c78cf591dff44fbea1af95baec4fc3eee5f327edb62fabc91ae3" gracePeriod=30
Sep 11 18:24:52.544603 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:52.542866    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p"]
Sep 11 18:24:52.544603 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:52.543100    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p" podUID="2b2cd515-f0cd-449b-a5f4-4eea1c9ece71" containerName="payload-pre-processing" containerID="cri-o://6e1ff1303cdbaa5b5e4edf7bbdf36ddb69e988e6848e9df2029ad7d873df1ebc" gracePeriod=30
Sep 11 18:24:52.812345 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:52.812320    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9"
Sep 11 18:24:52.840262 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:52.840230    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p"
Sep 11 18:24:52.911894 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:52.911859    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0558cdf-cfd6-4f72-b450-1a4a29b1f87b-plugins-config-volume\") pod \"d0558cdf-cfd6-4f72-b450-1a4a29b1f87b\" (UID: \"d0558cdf-cfd6-4f72-b450-1a4a29b1f87b\") "
Sep 11 18:24:52.912133 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:52.911960    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xv88g\" (UniqueName: \"kubernetes.io/projected/2b2cd515-f0cd-449b-a5f4-4eea1c9ece71-kube-api-access-xv88g\") pod \"2b2cd515-f0cd-449b-a5f4-4eea1c9ece71\" (UID: \"2b2cd515-f0cd-449b-a5f4-4eea1c9ece71\") "
Sep 11 18:24:52.912133 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:52.911979    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b2cd515-f0cd-449b-a5f4-4eea1c9ece71-plugins-config-volume\") pod \"2b2cd515-f0cd-449b-a5f4-4eea1c9ece71\" (UID: \"2b2cd515-f0cd-449b-a5f4-4eea1c9ece71\") "
Sep 11 18:24:52.912133 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:52.912015    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6rwz\" (UniqueName: \"kubernetes.io/projected/d0558cdf-cfd6-4f72-b450-1a4a29b1f87b-kube-api-access-q6rwz\") pod \"d0558cdf-cfd6-4f72-b450-1a4a29b1f87b\" (UID: \"d0558cdf-cfd6-4f72-b450-1a4a29b1f87b\") "
Sep 11 18:24:52.912373 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:52.912345    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b2cd515-f0cd-449b-a5f4-4eea1c9ece71-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2b2cd515-f0cd-449b-a5f4-4eea1c9ece71" (UID: "2b2cd515-f0cd-449b-a5f4-4eea1c9ece71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:24:52.912426 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:52.912350    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0558cdf-cfd6-4f72-b450-1a4a29b1f87b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d0558cdf-cfd6-4f72-b450-1a4a29b1f87b" (UID: "d0558cdf-cfd6-4f72-b450-1a4a29b1f87b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:24:52.914234 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:52.914207    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0558cdf-cfd6-4f72-b450-1a4a29b1f87b-kube-api-access-q6rwz" (OuterVolumeSpecName: "kube-api-access-q6rwz") pod "d0558cdf-cfd6-4f72-b450-1a4a29b1f87b" (UID: "d0558cdf-cfd6-4f72-b450-1a4a29b1f87b"). InnerVolumeSpecName "kube-api-access-q6rwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:24:52.914343 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:52.914241    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2cd515-f0cd-449b-a5f4-4eea1c9ece71-kube-api-access-xv88g" (OuterVolumeSpecName: "kube-api-access-xv88g") pod "2b2cd515-f0cd-449b-a5f4-4eea1c9ece71" (UID: "2b2cd515-f0cd-449b-a5f4-4eea1c9ece71"). InnerVolumeSpecName "kube-api-access-xv88g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:24:53.012866 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:53.012825    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6rwz\" (UniqueName: \"kubernetes.io/projected/d0558cdf-cfd6-4f72-b450-1a4a29b1f87b-kube-api-access-q6rwz\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:24:53.012866 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:53.012858    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0558cdf-cfd6-4f72-b450-1a4a29b1f87b-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:24:53.012866 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:53.012869    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv88g\" (UniqueName: \"kubernetes.io/projected/2b2cd515-f0cd-449b-a5f4-4eea1c9ece71-kube-api-access-xv88g\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:24:53.012866 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:53.012879    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b2cd515-f0cd-449b-a5f4-4eea1c9ece71-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:24:53.546380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:53.546343    2576 generic.go:356] "Generic (PLEG): container finished" podID="d0558cdf-cfd6-4f72-b450-1a4a29b1f87b" containerID="12a4283bc3a1c78cf591dff44fbea1af95baec4fc3eee5f327edb62fabc91ae3" exitCode=0
Sep 11 18:24:53.546865 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:53.546385    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9"
Sep 11 18:24:53.546865 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:53.546417    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9" event={"ID":"d0558cdf-cfd6-4f72-b450-1a4a29b1f87b","Type":"ContainerDied","Data":"12a4283bc3a1c78cf591dff44fbea1af95baec4fc3eee5f327edb62fabc91ae3"}
Sep 11 18:24:53.546865 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:53.546474    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9" event={"ID":"d0558cdf-cfd6-4f72-b450-1a4a29b1f87b","Type":"ContainerDied","Data":"cefec8b5ff78592ad136d457e9bba08d5c8c5e1fdf8c2a5fe49e6d680fb366a4"}
Sep 11 18:24:53.546865 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:53.546502    2576 scope.go:117] "RemoveContainer" containerID="12a4283bc3a1c78cf591dff44fbea1af95baec4fc3eee5f327edb62fabc91ae3"
Sep 11 18:24:53.548280 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:53.548251    2576 generic.go:356] "Generic (PLEG): container finished" podID="2b2cd515-f0cd-449b-a5f4-4eea1c9ece71" containerID="6e1ff1303cdbaa5b5e4edf7bbdf36ddb69e988e6848e9df2029ad7d873df1ebc" exitCode=0
Sep 11 18:24:53.548403 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:53.548296    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p"
Sep 11 18:24:53.548403 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:53.548344    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p" event={"ID":"2b2cd515-f0cd-449b-a5f4-4eea1c9ece71","Type":"ContainerDied","Data":"6e1ff1303cdbaa5b5e4edf7bbdf36ddb69e988e6848e9df2029ad7d873df1ebc"}
Sep 11 18:24:53.548516 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:53.548409    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p" event={"ID":"2b2cd515-f0cd-449b-a5f4-4eea1c9ece71","Type":"ContainerDied","Data":"138a732b740213e05f3b2fcc01bad5b78519f72dd01230f660e17ee59091a400"}
Sep 11 18:24:53.556965 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:53.556930    2576 scope.go:117] "RemoveContainer" containerID="12a4283bc3a1c78cf591dff44fbea1af95baec4fc3eee5f327edb62fabc91ae3"
Sep 11 18:24:53.557354 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:24:53.557323    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12a4283bc3a1c78cf591dff44fbea1af95baec4fc3eee5f327edb62fabc91ae3\": container with ID starting with 12a4283bc3a1c78cf591dff44fbea1af95baec4fc3eee5f327edb62fabc91ae3 not found: ID does not exist" containerID="12a4283bc3a1c78cf591dff44fbea1af95baec4fc3eee5f327edb62fabc91ae3"
Sep 11 18:24:53.557449 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:53.557363    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12a4283bc3a1c78cf591dff44fbea1af95baec4fc3eee5f327edb62fabc91ae3"} err="failed to get container status \"12a4283bc3a1c78cf591dff44fbea1af95baec4fc3eee5f327edb62fabc91ae3\": rpc error: code = NotFound desc = could not find container \"12a4283bc3a1c78cf591dff44fbea1af95baec4fc3eee5f327edb62fabc91ae3\": container with ID starting with 12a4283bc3a1c78cf591dff44fbea1af95baec4fc3eee5f327edb62fabc91ae3 not found: ID does not exist"
Sep 11 18:24:53.557449 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:53.557384    2576 scope.go:117] "RemoveContainer" containerID="6e1ff1303cdbaa5b5e4edf7bbdf36ddb69e988e6848e9df2029ad7d873df1ebc"
Sep 11 18:24:53.565876 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:53.565858    2576 scope.go:117] "RemoveContainer" containerID="6e1ff1303cdbaa5b5e4edf7bbdf36ddb69e988e6848e9df2029ad7d873df1ebc"
Sep 11 18:24:53.566176 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:24:53.566156    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e1ff1303cdbaa5b5e4edf7bbdf36ddb69e988e6848e9df2029ad7d873df1ebc\": container with ID starting with 6e1ff1303cdbaa5b5e4edf7bbdf36ddb69e988e6848e9df2029ad7d873df1ebc not found: ID does not exist" containerID="6e1ff1303cdbaa5b5e4edf7bbdf36ddb69e988e6848e9df2029ad7d873df1ebc"
Sep 11 18:24:53.566249 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:53.566185    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e1ff1303cdbaa5b5e4edf7bbdf36ddb69e988e6848e9df2029ad7d873df1ebc"} err="failed to get container status \"6e1ff1303cdbaa5b5e4edf7bbdf36ddb69e988e6848e9df2029ad7d873df1ebc\": rpc error: code = NotFound desc = could not find container \"6e1ff1303cdbaa5b5e4edf7bbdf36ddb69e988e6848e9df2029ad7d873df1ebc\": container with ID starting with 6e1ff1303cdbaa5b5e4edf7bbdf36ddb69e988e6848e9df2029ad7d873df1ebc not found: ID does not exist"
Sep 11 18:24:53.580598 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:53.580525    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9"]
Sep 11 18:24:53.587848 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:53.587821    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f8c01007-677cb74fcd-59ck9"]
Sep 11 18:24:53.603397 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:53.603344    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p"]
Sep 11 18:24:53.605356 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:53.605331    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f8c01007-7758f9dfb6-4879p"]
Sep 11 18:24:53.910572 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:53.910488    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b2cd515-f0cd-449b-a5f4-4eea1c9ece71" path="/var/lib/kubelet/pods/2b2cd515-f0cd-449b-a5f4-4eea1c9ece71/volumes"
Sep 11 18:24:53.910925 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:53.910910    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0558cdf-cfd6-4f72-b450-1a4a29b1f87b" path="/var/lib/kubelet/pods/d0558cdf-cfd6-4f72-b450-1a4a29b1f87b/volumes"
Sep 11 18:24:54.687324 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.687292    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"]
Sep 11 18:24:54.687753 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.687675    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0558cdf-cfd6-4f72-b450-1a4a29b1f87b" containerName="payload-processing"
Sep 11 18:24:54.687753 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.687687    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0558cdf-cfd6-4f72-b450-1a4a29b1f87b" containerName="payload-processing"
Sep 11 18:24:54.687753 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.687701    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b2cd515-f0cd-449b-a5f4-4eea1c9ece71" containerName="payload-pre-processing"
Sep 11 18:24:54.687753 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.687706    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b2cd515-f0cd-449b-a5f4-4eea1c9ece71" containerName="payload-pre-processing"
Sep 11 18:24:54.687901 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.687762    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0558cdf-cfd6-4f72-b450-1a4a29b1f87b" containerName="payload-processing"
Sep 11 18:24:54.687901 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.687773    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b2cd515-f0cd-449b-a5f4-4eea1c9ece71" containerName="payload-pre-processing"
Sep 11 18:24:54.692345 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.692324    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:24:54.694885 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.694855    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-553596\"/\"kube-root-ca.crt\""
Sep 11 18:24:54.695866 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.695847    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-553596\"/\"openshift-service-ca.crt\""
Sep 11 18:24:54.695976 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.695883    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-553596\"/\"default-dockercfg-j5lh6\""
Sep 11 18:24:54.695976 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.695887    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-553596\"/\"simulator-w4-553596-kserve-self-signed-certs\""
Sep 11 18:24:54.703194 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.703168    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"]
Sep 11 18:24:54.832069 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.832028    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f798c6d-ed14-4a83-b321-5329ae4d7286-tls-certs\") pod \"simulator-w4-553596-kserve-7c74f776f4-4z7zq\" (UID: \"2f798c6d-ed14-4a83-b321-5329ae4d7286\") " pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:24:54.832069 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.832073    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f798c6d-ed14-4a83-b321-5329ae4d7286-dshm\") pod \"simulator-w4-553596-kserve-7c74f776f4-4z7zq\" (UID: \"2f798c6d-ed14-4a83-b321-5329ae4d7286\") " pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:24:54.832283 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.832151    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f798c6d-ed14-4a83-b321-5329ae4d7286-model-cache\") pod \"simulator-w4-553596-kserve-7c74f776f4-4z7zq\" (UID: \"2f798c6d-ed14-4a83-b321-5329ae4d7286\") " pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:24:54.832283 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.832211    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jknbx\" (UniqueName: \"kubernetes.io/projected/2f798c6d-ed14-4a83-b321-5329ae4d7286-kube-api-access-jknbx\") pod \"simulator-w4-553596-kserve-7c74f776f4-4z7zq\" (UID: \"2f798c6d-ed14-4a83-b321-5329ae4d7286\") " pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:24:54.832283 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.832248    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f798c6d-ed14-4a83-b321-5329ae4d7286-tmp-dir\") pod \"simulator-w4-553596-kserve-7c74f776f4-4z7zq\" (UID: \"2f798c6d-ed14-4a83-b321-5329ae4d7286\") " pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:24:54.832283 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.832274    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f798c6d-ed14-4a83-b321-5329ae4d7286-home\") pod \"simulator-w4-553596-kserve-7c74f776f4-4z7zq\" (UID: \"2f798c6d-ed14-4a83-b321-5329ae4d7286\") " pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:24:54.933220 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.933161    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jknbx\" (UniqueName: \"kubernetes.io/projected/2f798c6d-ed14-4a83-b321-5329ae4d7286-kube-api-access-jknbx\") pod \"simulator-w4-553596-kserve-7c74f776f4-4z7zq\" (UID: \"2f798c6d-ed14-4a83-b321-5329ae4d7286\") " pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:24:54.933220 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.933229    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f798c6d-ed14-4a83-b321-5329ae4d7286-tmp-dir\") pod \"simulator-w4-553596-kserve-7c74f776f4-4z7zq\" (UID: \"2f798c6d-ed14-4a83-b321-5329ae4d7286\") " pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:24:54.933491 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.933251    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f798c6d-ed14-4a83-b321-5329ae4d7286-home\") pod \"simulator-w4-553596-kserve-7c74f776f4-4z7zq\" (UID: \"2f798c6d-ed14-4a83-b321-5329ae4d7286\") " pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:24:54.933491 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.933295    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f798c6d-ed14-4a83-b321-5329ae4d7286-tls-certs\") pod \"simulator-w4-553596-kserve-7c74f776f4-4z7zq\" (UID: \"2f798c6d-ed14-4a83-b321-5329ae4d7286\") " pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:24:54.933491 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.933316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f798c6d-ed14-4a83-b321-5329ae4d7286-dshm\") pod \"simulator-w4-553596-kserve-7c74f776f4-4z7zq\" (UID: \"2f798c6d-ed14-4a83-b321-5329ae4d7286\") " pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:24:54.933491 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.933402    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f798c6d-ed14-4a83-b321-5329ae4d7286-model-cache\") pod \"simulator-w4-553596-kserve-7c74f776f4-4z7zq\" (UID: \"2f798c6d-ed14-4a83-b321-5329ae4d7286\") " pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:24:54.933756 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.933732    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f798c6d-ed14-4a83-b321-5329ae4d7286-home\") pod \"simulator-w4-553596-kserve-7c74f776f4-4z7zq\" (UID: \"2f798c6d-ed14-4a83-b321-5329ae4d7286\") " pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:24:54.933794 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.933748    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f798c6d-ed14-4a83-b321-5329ae4d7286-tmp-dir\") pod \"simulator-w4-553596-kserve-7c74f776f4-4z7zq\" (UID: \"2f798c6d-ed14-4a83-b321-5329ae4d7286\") " pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:24:54.933858 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.933842    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f798c6d-ed14-4a83-b321-5329ae4d7286-model-cache\") pod \"simulator-w4-553596-kserve-7c74f776f4-4z7zq\" (UID: \"2f798c6d-ed14-4a83-b321-5329ae4d7286\") " pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:24:54.935685 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.935657    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f798c6d-ed14-4a83-b321-5329ae4d7286-dshm\") pod \"simulator-w4-553596-kserve-7c74f776f4-4z7zq\" (UID: \"2f798c6d-ed14-4a83-b321-5329ae4d7286\") " pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:24:54.935935 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.935918    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f798c6d-ed14-4a83-b321-5329ae4d7286-tls-certs\") pod \"simulator-w4-553596-kserve-7c74f776f4-4z7zq\" (UID: \"2f798c6d-ed14-4a83-b321-5329ae4d7286\") " pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:24:54.942143 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:54.942111    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jknbx\" (UniqueName: \"kubernetes.io/projected/2f798c6d-ed14-4a83-b321-5329ae4d7286-kube-api-access-jknbx\") pod \"simulator-w4-553596-kserve-7c74f776f4-4z7zq\" (UID: \"2f798c6d-ed14-4a83-b321-5329ae4d7286\") " pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:24:55.002170 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:55.002127    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:24:55.279801 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:55.279728    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"]
Sep 11 18:24:55.282643 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:24:55.282616    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f798c6d_ed14_4a83_b321_5329ae4d7286.slice/crio-7faf8bb34381491abb0f9a19089ef1549a0cc46b450964a5069b89524037aca9 WatchSource:0}: Error finding container 7faf8bb34381491abb0f9a19089ef1549a0cc46b450964a5069b89524037aca9: Status 404 returned error can't find the container with id 7faf8bb34381491abb0f9a19089ef1549a0cc46b450964a5069b89524037aca9
Sep 11 18:24:55.562710 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:55.562613    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq" event={"ID":"2f798c6d-ed14-4a83-b321-5329ae4d7286","Type":"ContainerStarted","Data":"966d292d8eb1d942aa04e160e0a9cca87658a6e59a721e67b6eab498755d6264"}
Sep 11 18:24:55.562710 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:55.562659    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq" event={"ID":"2f798c6d-ed14-4a83-b321-5329ae4d7286","Type":"ContainerStarted","Data":"7faf8bb34381491abb0f9a19089ef1549a0cc46b450964a5069b89524037aca9"}
Sep 11 18:24:55.582944 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:55.582892    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq" podStartSLOduration=1.582874261 podStartE2EDuration="1.582874261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:24:55.581120426 +0000 UTC m=+894.182481590" watchObservedRunningTime="2026-09-11 18:24:55.582874261 +0000 UTC m=+894.184235423"
Sep 11 18:24:56.000891 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:56.000856    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:24:58.601461 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:58.601424    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9"]
Sep 11 18:24:58.604711 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:58.604693    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9"
Sep 11 18:24:58.606905 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:58.606882    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ed21d135-dockercfg-km74q\""
Sep 11 18:24:58.607021 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:58.606961    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ed21d135\""
Sep 11 18:24:58.615443 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:58.615418    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9"]
Sep 11 18:24:58.771450 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:58.771404    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kjfd\" (UniqueName: \"kubernetes.io/projected/09212d45-c629-4934-926f-37ae6945f2a5-kube-api-access-4kjfd\") pod \"payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9\" (UID: \"09212d45-c629-4934-926f-37ae6945f2a5\") " pod="openshift-ingress/payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9"
Sep 11 18:24:58.771668 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:58.771469    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09212d45-c629-4934-926f-37ae6945f2a5-plugins-config-volume\") pod \"payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9\" (UID: \"09212d45-c629-4934-926f-37ae6945f2a5\") " pod="openshift-ingress/payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9"
Sep 11 18:24:58.872437 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:58.872333    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kjfd\" (UniqueName: \"kubernetes.io/projected/09212d45-c629-4934-926f-37ae6945f2a5-kube-api-access-4kjfd\") pod \"payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9\" (UID: \"09212d45-c629-4934-926f-37ae6945f2a5\") " pod="openshift-ingress/payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9"
Sep 11 18:24:58.872437 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:58.872378    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09212d45-c629-4934-926f-37ae6945f2a5-plugins-config-volume\") pod \"payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9\" (UID: \"09212d45-c629-4934-926f-37ae6945f2a5\") " pod="openshift-ingress/payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9"
Sep 11 18:24:58.873131 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:58.873105    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09212d45-c629-4934-926f-37ae6945f2a5-plugins-config-volume\") pod \"payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9\" (UID: \"09212d45-c629-4934-926f-37ae6945f2a5\") " pod="openshift-ingress/payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9"
Sep 11 18:24:58.880710 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:58.880682    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kjfd\" (UniqueName: \"kubernetes.io/projected/09212d45-c629-4934-926f-37ae6945f2a5-kube-api-access-4kjfd\") pod \"payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9\" (UID: \"09212d45-c629-4934-926f-37ae6945f2a5\") " pod="openshift-ingress/payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9"
Sep 11 18:24:58.915316 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:58.915280    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9"
Sep 11 18:24:59.192901 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:59.190911    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9"]
Sep 11 18:24:59.196385 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:24:59.196356    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09212d45_c629_4934_926f_37ae6945f2a5.slice/crio-b2446bc757e3ca8e0f2d1c92079ea0b07d04d5c0aeb3962cba0cfe749f9361f9 WatchSource:0}: Error finding container b2446bc757e3ca8e0f2d1c92079ea0b07d04d5c0aeb3962cba0cfe749f9361f9: Status 404 returned error can't find the container with id b2446bc757e3ca8e0f2d1c92079ea0b07d04d5c0aeb3962cba0cfe749f9361f9
Sep 11 18:24:59.583633 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:59.583521    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9" event={"ID":"09212d45-c629-4934-926f-37ae6945f2a5","Type":"ContainerStarted","Data":"82ba30f7cd80a465a1ce7dd6a5cef7dd584e690ca5d9cc478ea7a7e7bae1880f"}
Sep 11 18:24:59.583633 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:59.583596    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9" event={"ID":"09212d45-c629-4934-926f-37ae6945f2a5","Type":"ContainerStarted","Data":"b2446bc757e3ca8e0f2d1c92079ea0b07d04d5c0aeb3962cba0cfe749f9361f9"}
Sep 11 18:24:59.583867 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:59.583639    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9"
Sep 11 18:24:59.599074 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:24:59.599022    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9" podStartSLOduration=1.599006841 podStartE2EDuration="1.599006841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:24:59.598952983 +0000 UTC m=+898.200314149" watchObservedRunningTime="2026-09-11 18:24:59.599006841 +0000 UTC m=+898.200368001"
Sep 11 18:25:05.002676 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:05.002607    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:25:05.002676 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:05.002685    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:25:05.015541 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:05.015515    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:25:05.622784 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:05.622753    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:25:10.590756 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:10.590725    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9"
Sep 11 18:25:16.118213 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:16.118174    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:25:35.614352 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:35.614315    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:25:38.199063 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:38.199026    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9"]
Sep 11 18:25:38.199489 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:38.199287    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9" podUID="09212d45-c629-4934-926f-37ae6945f2a5" containerName="payload-processing" containerID="cri-o://82ba30f7cd80a465a1ce7dd6a5cef7dd584e690ca5d9cc478ea7a7e7bae1880f" gracePeriod=30
Sep 11 18:25:38.449970 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:38.449910    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9"
Sep 11 18:25:38.533869 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:38.533837    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kjfd\" (UniqueName: \"kubernetes.io/projected/09212d45-c629-4934-926f-37ae6945f2a5-kube-api-access-4kjfd\") pod \"09212d45-c629-4934-926f-37ae6945f2a5\" (UID: \"09212d45-c629-4934-926f-37ae6945f2a5\") "
Sep 11 18:25:38.533869 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:38.533879    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09212d45-c629-4934-926f-37ae6945f2a5-plugins-config-volume\") pod \"09212d45-c629-4934-926f-37ae6945f2a5\" (UID: \"09212d45-c629-4934-926f-37ae6945f2a5\") "
Sep 11 18:25:38.534270 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:38.534239    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09212d45-c629-4934-926f-37ae6945f2a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "09212d45-c629-4934-926f-37ae6945f2a5" (UID: "09212d45-c629-4934-926f-37ae6945f2a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:25:38.536002 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:38.535968    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09212d45-c629-4934-926f-37ae6945f2a5-kube-api-access-4kjfd" (OuterVolumeSpecName: "kube-api-access-4kjfd") pod "09212d45-c629-4934-926f-37ae6945f2a5" (UID: "09212d45-c629-4934-926f-37ae6945f2a5"). InnerVolumeSpecName "kube-api-access-4kjfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:25:38.635023 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:38.634981    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kjfd\" (UniqueName: \"kubernetes.io/projected/09212d45-c629-4934-926f-37ae6945f2a5-kube-api-access-4kjfd\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:25:38.635023 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:38.635018    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09212d45-c629-4934-926f-37ae6945f2a5-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:25:38.766371 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:38.766296    2576 generic.go:356] "Generic (PLEG): container finished" podID="09212d45-c629-4934-926f-37ae6945f2a5" containerID="82ba30f7cd80a465a1ce7dd6a5cef7dd584e690ca5d9cc478ea7a7e7bae1880f" exitCode=0
Sep 11 18:25:38.766371 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:38.766342    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9"
Sep 11 18:25:38.766371 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:38.766360    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9" event={"ID":"09212d45-c629-4934-926f-37ae6945f2a5","Type":"ContainerDied","Data":"82ba30f7cd80a465a1ce7dd6a5cef7dd584e690ca5d9cc478ea7a7e7bae1880f"}
Sep 11 18:25:38.766630 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:38.766403    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9" event={"ID":"09212d45-c629-4934-926f-37ae6945f2a5","Type":"ContainerDied","Data":"b2446bc757e3ca8e0f2d1c92079ea0b07d04d5c0aeb3962cba0cfe749f9361f9"}
Sep 11 18:25:38.766630 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:38.766426    2576 scope.go:117] "RemoveContainer" containerID="82ba30f7cd80a465a1ce7dd6a5cef7dd584e690ca5d9cc478ea7a7e7bae1880f"
Sep 11 18:25:38.777339 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:38.777320    2576 scope.go:117] "RemoveContainer" containerID="82ba30f7cd80a465a1ce7dd6a5cef7dd584e690ca5d9cc478ea7a7e7bae1880f"
Sep 11 18:25:38.777662 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:25:38.777643    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82ba30f7cd80a465a1ce7dd6a5cef7dd584e690ca5d9cc478ea7a7e7bae1880f\": container with ID starting with 82ba30f7cd80a465a1ce7dd6a5cef7dd584e690ca5d9cc478ea7a7e7bae1880f not found: ID does not exist" containerID="82ba30f7cd80a465a1ce7dd6a5cef7dd584e690ca5d9cc478ea7a7e7bae1880f"
Sep 11 18:25:38.777731 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:38.777676    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82ba30f7cd80a465a1ce7dd6a5cef7dd584e690ca5d9cc478ea7a7e7bae1880f"} err="failed to get container status \"82ba30f7cd80a465a1ce7dd6a5cef7dd584e690ca5d9cc478ea7a7e7bae1880f\": rpc error: code = NotFound desc = could not find container \"82ba30f7cd80a465a1ce7dd6a5cef7dd584e690ca5d9cc478ea7a7e7bae1880f\": container with ID starting with 82ba30f7cd80a465a1ce7dd6a5cef7dd584e690ca5d9cc478ea7a7e7bae1880f not found: ID does not exist"
Sep 11 18:25:38.808292 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:38.808262    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9"]
Sep 11 18:25:38.811971 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:38.811948    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ed21d135-5b89f8cc8f-9gfb9"]
Sep 11 18:25:39.916739 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:39.916701    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09212d45-c629-4934-926f-37ae6945f2a5" path="/var/lib/kubelet/pods/09212d45-c629-4934-926f-37ae6945f2a5/volumes"
Sep 11 18:25:47.649458 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:47.649418    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52"]
Sep 11 18:25:47.650084 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:47.649952    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09212d45-c629-4934-926f-37ae6945f2a5" containerName="payload-processing"
Sep 11 18:25:47.650084 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:47.650009    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="09212d45-c629-4934-926f-37ae6945f2a5" containerName="payload-processing"
Sep 11 18:25:47.650206 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:47.650123    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="09212d45-c629-4934-926f-37ae6945f2a5" containerName="payload-processing"
Sep 11 18:25:47.654497 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:47.654476    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52"
Sep 11 18:25:47.656886 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:47.656862    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-23f49f11\""
Sep 11 18:25:47.657052 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:47.657034    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-23f49f11-dockercfg-lg5h2\""
Sep 11 18:25:47.663671 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:47.663644    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52"]
Sep 11 18:25:47.693279 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:47.693237    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw"]
Sep 11 18:25:47.696574 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:47.696550    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw"
Sep 11 18:25:47.707289 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:47.707266    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw"]
Sep 11 18:25:47.711986 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:47.711963    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95f2r\" (UniqueName: \"kubernetes.io/projected/2e144939-11fa-4ba0-b74c-900e823665f9-kube-api-access-95f2r\") pod \"payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52\" (UID: \"2e144939-11fa-4ba0-b74c-900e823665f9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52"
Sep 11 18:25:47.712112 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:47.712075    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e144939-11fa-4ba0-b74c-900e823665f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52\" (UID: \"2e144939-11fa-4ba0-b74c-900e823665f9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52"
Sep 11 18:25:47.712181 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:47.712134    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58t7n\" (UniqueName: \"kubernetes.io/projected/c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe-kube-api-access-58t7n\") pod \"payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw\" (UID: \"c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe\") " pod="openshift-ingress/payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw"
Sep 11 18:25:47.712181 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:47.712170    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe-plugins-config-volume\") pod \"payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw\" (UID: \"c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe\") " pod="openshift-ingress/payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw"
Sep 11 18:25:47.812769 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:47.812735    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95f2r\" (UniqueName: \"kubernetes.io/projected/2e144939-11fa-4ba0-b74c-900e823665f9-kube-api-access-95f2r\") pod \"payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52\" (UID: \"2e144939-11fa-4ba0-b74c-900e823665f9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52"
Sep 11 18:25:47.812948 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:47.812829    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e144939-11fa-4ba0-b74c-900e823665f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52\" (UID: \"2e144939-11fa-4ba0-b74c-900e823665f9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52"
Sep 11 18:25:47.812948 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:47.812873    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58t7n\" (UniqueName: \"kubernetes.io/projected/c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe-kube-api-access-58t7n\") pod \"payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw\" (UID: \"c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe\") " pod="openshift-ingress/payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw"
Sep 11 18:25:47.812948 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:47.812908    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe-plugins-config-volume\") pod \"payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw\" (UID: \"c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe\") " pod="openshift-ingress/payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw"
Sep 11 18:25:47.813661 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:47.813642    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe-plugins-config-volume\") pod \"payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw\" (UID: \"c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe\") " pod="openshift-ingress/payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw"
Sep 11 18:25:47.813718 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:47.813646    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e144939-11fa-4ba0-b74c-900e823665f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52\" (UID: \"2e144939-11fa-4ba0-b74c-900e823665f9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52"
Sep 11 18:25:47.820479 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:47.820452    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58t7n\" (UniqueName: \"kubernetes.io/projected/c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe-kube-api-access-58t7n\") pod \"payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw\" (UID: \"c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe\") " pod="openshift-ingress/payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw"
Sep 11 18:25:47.820479 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:47.820471    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95f2r\" (UniqueName: \"kubernetes.io/projected/2e144939-11fa-4ba0-b74c-900e823665f9-kube-api-access-95f2r\") pod \"payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52\" (UID: \"2e144939-11fa-4ba0-b74c-900e823665f9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52"
Sep 11 18:25:47.965468 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:47.965436    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52"
Sep 11 18:25:48.007678 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.007645    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw"
Sep 11 18:25:48.234441 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.234416    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52"]
Sep 11 18:25:48.235890 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:25:48.235863    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e144939_11fa_4ba0_b74c_900e823665f9.slice/crio-1951fe26640116fd44c4b0e273a03b36a5f68770755b0cc6440f401a1e9ce95f WatchSource:0}: Error finding container 1951fe26640116fd44c4b0e273a03b36a5f68770755b0cc6440f401a1e9ce95f: Status 404 returned error can't find the container with id 1951fe26640116fd44c4b0e273a03b36a5f68770755b0cc6440f401a1e9ce95f
Sep 11 18:25:48.289212 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.289190    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw"]
Sep 11 18:25:48.290937 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:25:48.290913    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e04c71_3e09_4ef7_8a7e_6bfe54e05ffe.slice/crio-cfe6cffed2c6d5fc7d069cdc2abc666232cfe12506a7e4f3e546cfbd9fa527d0 WatchSource:0}: Error finding container cfe6cffed2c6d5fc7d069cdc2abc666232cfe12506a7e4f3e546cfbd9fa527d0: Status 404 returned error can't find the container with id cfe6cffed2c6d5fc7d069cdc2abc666232cfe12506a7e4f3e546cfbd9fa527d0
Sep 11 18:25:48.477963 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.477928    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"]
Sep 11 18:25:48.481256 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.481236    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:25:48.483527 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.483464    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f63893\"/\"kube-root-ca.crt\""
Sep 11 18:25:48.484273 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.484254    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f63893\"/\"openshift-service-ca.crt\""
Sep 11 18:25:48.484566 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.484552    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f63893\"/\"default-dockercfg-rthkn\""
Sep 11 18:25:48.484684 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.484629    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f63893\"/\"test-model-f63893-kserve-self-signed-certs\""
Sep 11 18:25:48.491744 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.491725    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"]
Sep 11 18:25:48.517660 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.517631    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqp6h\" (UniqueName: \"kubernetes.io/projected/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-kube-api-access-nqp6h\") pod \"test-model-f63893-kserve-7878589b9f-w5v7g\" (UID: \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\") " pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:25:48.517782 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.517663    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-tls-certs\") pod \"test-model-f63893-kserve-7878589b9f-w5v7g\" (UID: \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\") " pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:25:48.517782 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.517693    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-dshm\") pod \"test-model-f63893-kserve-7878589b9f-w5v7g\" (UID: \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\") " pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:25:48.517782 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.517755    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-home\") pod \"test-model-f63893-kserve-7878589b9f-w5v7g\" (UID: \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\") " pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:25:48.517782 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.517778    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-tmp-dir\") pod \"test-model-f63893-kserve-7878589b9f-w5v7g\" (UID: \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\") " pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:25:48.517994 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.517852    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-model-cache\") pod \"test-model-f63893-kserve-7878589b9f-w5v7g\" (UID: \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\") " pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:25:48.619065 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.619028    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-dshm\") pod \"test-model-f63893-kserve-7878589b9f-w5v7g\" (UID: \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\") " pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:25:48.619254 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.619076    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-home\") pod \"test-model-f63893-kserve-7878589b9f-w5v7g\" (UID: \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\") " pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:25:48.619254 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.619099    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-tmp-dir\") pod \"test-model-f63893-kserve-7878589b9f-w5v7g\" (UID: \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\") " pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:25:48.619254 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.619133    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-model-cache\") pod \"test-model-f63893-kserve-7878589b9f-w5v7g\" (UID: \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\") " pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:25:48.619254 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.619190    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqp6h\" (UniqueName: \"kubernetes.io/projected/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-kube-api-access-nqp6h\") pod \"test-model-f63893-kserve-7878589b9f-w5v7g\" (UID: \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\") " pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:25:48.619442 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.619312    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-tls-certs\") pod \"test-model-f63893-kserve-7878589b9f-w5v7g\" (UID: \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\") " pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:25:48.619596 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.619558    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-home\") pod \"test-model-f63893-kserve-7878589b9f-w5v7g\" (UID: \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\") " pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:25:48.619674 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.619654    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-tmp-dir\") pod \"test-model-f63893-kserve-7878589b9f-w5v7g\" (UID: \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\") " pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:25:48.619718 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.619689    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-model-cache\") pod \"test-model-f63893-kserve-7878589b9f-w5v7g\" (UID: \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\") " pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:25:48.621490 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.621462    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-dshm\") pod \"test-model-f63893-kserve-7878589b9f-w5v7g\" (UID: \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\") " pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:25:48.621756 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.621738    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-tls-certs\") pod \"test-model-f63893-kserve-7878589b9f-w5v7g\" (UID: \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\") " pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:25:48.626765 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.626747    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqp6h\" (UniqueName: \"kubernetes.io/projected/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-kube-api-access-nqp6h\") pod \"test-model-f63893-kserve-7878589b9f-w5v7g\" (UID: \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\") " pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:25:48.790953 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.790855    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:25:48.816785 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.816745    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw" event={"ID":"c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe","Type":"ContainerStarted","Data":"2c205cf83299a288c84d3743cdc4cd14c1cef09e7d6fa6fccb32401a621aa323"}
Sep 11 18:25:48.817016 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.817000    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw" event={"ID":"c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe","Type":"ContainerStarted","Data":"cfe6cffed2c6d5fc7d069cdc2abc666232cfe12506a7e4f3e546cfbd9fa527d0"}
Sep 11 18:25:48.817129 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.817110    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw"
Sep 11 18:25:48.818532 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.818509    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52" event={"ID":"2e144939-11fa-4ba0-b74c-900e823665f9","Type":"ContainerStarted","Data":"0ae1d92443dea91d65599bef8e43d05b5b6a3f6ead595869b37b4bbb29d7ebaf"}
Sep 11 18:25:48.818639 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.818549    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52" event={"ID":"2e144939-11fa-4ba0-b74c-900e823665f9","Type":"ContainerStarted","Data":"1951fe26640116fd44c4b0e273a03b36a5f68770755b0cc6440f401a1e9ce95f"}
Sep 11 18:25:48.818741 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.818712    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52"
Sep 11 18:25:48.834421 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.834363    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw" podStartSLOduration=1.834344879 podStartE2EDuration="1.834344879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:48.834316015 +0000 UTC m=+947.435677183" watchObservedRunningTime="2026-09-11 18:25:48.834344879 +0000 UTC m=+947.435706043"
Sep 11 18:25:48.855143 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:48.854568    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52" podStartSLOduration=1.854545042 podStartE2EDuration="1.854545042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:48.853572214 +0000 UTC m=+947.454933386" watchObservedRunningTime="2026-09-11 18:25:48.854545042 +0000 UTC m=+947.455906206"
Sep 11 18:25:49.071006 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:49.070974    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"]
Sep 11 18:25:49.072662 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:25:49.072629    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38a9a26d_5f5a_4cf6_928a_cf5be2e04e40.slice/crio-6f8f826473b26b9e2533aa21c116c7787dac60597898f53f2fb569a2e331f385 WatchSource:0}: Error finding container 6f8f826473b26b9e2533aa21c116c7787dac60597898f53f2fb569a2e331f385: Status 404 returned error can't find the container with id 6f8f826473b26b9e2533aa21c116c7787dac60597898f53f2fb569a2e331f385
Sep 11 18:25:49.825192 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:49.825154    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g" event={"ID":"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40","Type":"ContainerStarted","Data":"85cb051e23254814dd0736005a1b41074907064710ea2364da8581eadde83266"}
Sep 11 18:25:49.825654 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:49.825203    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g" event={"ID":"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40","Type":"ContainerStarted","Data":"6f8f826473b26b9e2533aa21c116c7787dac60597898f53f2fb569a2e331f385"}
Sep 11 18:25:49.843593 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:49.843534    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g" podStartSLOduration=1.843507055 podStartE2EDuration="1.843507055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:49.843044126 +0000 UTC m=+948.444405293" watchObservedRunningTime="2026-09-11 18:25:49.843507055 +0000 UTC m=+948.444868222"
Sep 11 18:25:56.513412 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:56.513376    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:25:58.791123 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:58.791084    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:25:58.791123 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:58.791130    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:25:58.803862 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:58.803836    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:25:58.879168 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:58.879142    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:25:59.827150 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:59.827116    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52"
Sep 11 18:25:59.827722 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:25:59.827703    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw"
Sep 11 18:26:12.799909 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:12.799874    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:26:14.940858 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:14.940822    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb"]
Sep 11 18:26:14.941623 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:14.941547    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb" podUID="d5ba777e-d6e1-4b50-a97c-3155a80a2400" containerName="payload-processing" containerID="cri-o://9477f0feeec226cc6af56dec1d15a2b3bfa1c8530e6b0689fb8550ea4bcb67e6" gracePeriod=30
Sep 11 18:26:14.948955 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:14.948925    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc"]
Sep 11 18:26:14.949198 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:14.949150    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc" podUID="cb2a3574-ea6e-4b22-9aa8-f103e36a47c0" containerName="payload-pre-processing" containerID="cri-o://1f6eb0fdbaf18e327f3e34cff226d415d22b3442865f60d98dd007d7ea910a21" gracePeriod=30
Sep 11 18:26:15.123370 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.123328    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw"]
Sep 11 18:26:15.123672 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.123578    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw" podUID="c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe" containerName="payload-processing" containerID="cri-o://2c205cf83299a288c84d3743cdc4cd14c1cef09e7d6fa6fccb32401a621aa323" gracePeriod=30
Sep 11 18:26:15.129616 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.129574    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52"]
Sep 11 18:26:15.130934 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.129802    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52" podUID="2e144939-11fa-4ba0-b74c-900e823665f9" containerName="payload-pre-processing" containerID="cri-o://0ae1d92443dea91d65599bef8e43d05b5b6a3f6ead595869b37b4bbb29d7ebaf" gracePeriod=30
Sep 11 18:26:15.268388 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.268359    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc"
Sep 11 18:26:15.273411 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.273381    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb"
Sep 11 18:26:15.364730 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.364692    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb2a3574-ea6e-4b22-9aa8-f103e36a47c0-plugins-config-volume\") pod \"cb2a3574-ea6e-4b22-9aa8-f103e36a47c0\" (UID: \"cb2a3574-ea6e-4b22-9aa8-f103e36a47c0\") "
Sep 11 18:26:15.364930 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.364779    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ba777e-d6e1-4b50-a97c-3155a80a2400-plugins-config-volume\") pod \"d5ba777e-d6e1-4b50-a97c-3155a80a2400\" (UID: \"d5ba777e-d6e1-4b50-a97c-3155a80a2400\") "
Sep 11 18:26:15.364930 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.364840    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5ks8\" (UniqueName: \"kubernetes.io/projected/cb2a3574-ea6e-4b22-9aa8-f103e36a47c0-kube-api-access-m5ks8\") pod \"cb2a3574-ea6e-4b22-9aa8-f103e36a47c0\" (UID: \"cb2a3574-ea6e-4b22-9aa8-f103e36a47c0\") "
Sep 11 18:26:15.364930 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.364918    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6wp6\" (UniqueName: \"kubernetes.io/projected/d5ba777e-d6e1-4b50-a97c-3155a80a2400-kube-api-access-l6wp6\") pod \"d5ba777e-d6e1-4b50-a97c-3155a80a2400\" (UID: \"d5ba777e-d6e1-4b50-a97c-3155a80a2400\") "
Sep 11 18:26:15.365160 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.365112    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb2a3574-ea6e-4b22-9aa8-f103e36a47c0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cb2a3574-ea6e-4b22-9aa8-f103e36a47c0" (UID: "cb2a3574-ea6e-4b22-9aa8-f103e36a47c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:26:15.365499 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.365477    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5ba777e-d6e1-4b50-a97c-3155a80a2400-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d5ba777e-d6e1-4b50-a97c-3155a80a2400" (UID: "d5ba777e-d6e1-4b50-a97c-3155a80a2400"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:26:15.368674 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.368647    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb2a3574-ea6e-4b22-9aa8-f103e36a47c0-kube-api-access-m5ks8" (OuterVolumeSpecName: "kube-api-access-m5ks8") pod "cb2a3574-ea6e-4b22-9aa8-f103e36a47c0" (UID: "cb2a3574-ea6e-4b22-9aa8-f103e36a47c0"). InnerVolumeSpecName "kube-api-access-m5ks8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:26:15.369616 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.369593    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ba777e-d6e1-4b50-a97c-3155a80a2400-kube-api-access-l6wp6" (OuterVolumeSpecName: "kube-api-access-l6wp6") pod "d5ba777e-d6e1-4b50-a97c-3155a80a2400" (UID: "d5ba777e-d6e1-4b50-a97c-3155a80a2400"). InnerVolumeSpecName "kube-api-access-l6wp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:26:15.392794 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.392770    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw"
Sep 11 18:26:15.401129 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.401107    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52"
Sep 11 18:26:15.465935 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.465849    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95f2r\" (UniqueName: \"kubernetes.io/projected/2e144939-11fa-4ba0-b74c-900e823665f9-kube-api-access-95f2r\") pod \"2e144939-11fa-4ba0-b74c-900e823665f9\" (UID: \"2e144939-11fa-4ba0-b74c-900e823665f9\") "
Sep 11 18:26:15.466089 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.465985    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e144939-11fa-4ba0-b74c-900e823665f9-plugins-config-volume\") pod \"2e144939-11fa-4ba0-b74c-900e823665f9\" (UID: \"2e144939-11fa-4ba0-b74c-900e823665f9\") "
Sep 11 18:26:15.466089 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.466018    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58t7n\" (UniqueName: \"kubernetes.io/projected/c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe-kube-api-access-58t7n\") pod \"c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe\" (UID: \"c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe\") "
Sep 11 18:26:15.466089 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.466071    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe-plugins-config-volume\") pod \"c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe\" (UID: \"c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe\") "
Sep 11 18:26:15.466385 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.466358    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb2a3574-ea6e-4b22-9aa8-f103e36a47c0-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:26:15.466385 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.466384    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ba777e-d6e1-4b50-a97c-3155a80a2400-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:26:15.466534 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.466398    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5ks8\" (UniqueName: \"kubernetes.io/projected/cb2a3574-ea6e-4b22-9aa8-f103e36a47c0-kube-api-access-m5ks8\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:26:15.466534 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.466410    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6wp6\" (UniqueName: \"kubernetes.io/projected/d5ba777e-d6e1-4b50-a97c-3155a80a2400-kube-api-access-l6wp6\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:26:15.466534 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.466454    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe" (UID: "c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:26:15.466868 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.466842    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e144939-11fa-4ba0-b74c-900e823665f9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2e144939-11fa-4ba0-b74c-900e823665f9" (UID: "2e144939-11fa-4ba0-b74c-900e823665f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:26:15.469433 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.469408    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe-kube-api-access-58t7n" (OuterVolumeSpecName: "kube-api-access-58t7n") pod "c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe" (UID: "c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe"). InnerVolumeSpecName "kube-api-access-58t7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:26:15.470873 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.470848    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e144939-11fa-4ba0-b74c-900e823665f9-kube-api-access-95f2r" (OuterVolumeSpecName: "kube-api-access-95f2r") pod "2e144939-11fa-4ba0-b74c-900e823665f9" (UID: "2e144939-11fa-4ba0-b74c-900e823665f9"). InnerVolumeSpecName "kube-api-access-95f2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:26:15.572907 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.568281    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e144939-11fa-4ba0-b74c-900e823665f9-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:26:15.572907 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.568318    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58t7n\" (UniqueName: \"kubernetes.io/projected/c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe-kube-api-access-58t7n\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:26:15.572907 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.568334    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:26:15.572907 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.568349    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95f2r\" (UniqueName: \"kubernetes.io/projected/2e144939-11fa-4ba0-b74c-900e823665f9-kube-api-access-95f2r\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:26:15.950999 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.950960    2576 generic.go:356] "Generic (PLEG): container finished" podID="cb2a3574-ea6e-4b22-9aa8-f103e36a47c0" containerID="1f6eb0fdbaf18e327f3e34cff226d415d22b3442865f60d98dd007d7ea910a21" exitCode=0
Sep 11 18:26:15.951446 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.951030    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc"
Sep 11 18:26:15.951446 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.951041    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc" event={"ID":"cb2a3574-ea6e-4b22-9aa8-f103e36a47c0","Type":"ContainerDied","Data":"1f6eb0fdbaf18e327f3e34cff226d415d22b3442865f60d98dd007d7ea910a21"}
Sep 11 18:26:15.951446 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.951100    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc" event={"ID":"cb2a3574-ea6e-4b22-9aa8-f103e36a47c0","Type":"ContainerDied","Data":"8a649c151b0cc60c05414b5f85fa070977ca6f1f86311f987b4c63ef903103ea"}
Sep 11 18:26:15.951446 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.951141    2576 scope.go:117] "RemoveContainer" containerID="1f6eb0fdbaf18e327f3e34cff226d415d22b3442865f60d98dd007d7ea910a21"
Sep 11 18:26:15.952626 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.952417    2576 generic.go:356] "Generic (PLEG): container finished" podID="c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe" containerID="2c205cf83299a288c84d3743cdc4cd14c1cef09e7d6fa6fccb32401a621aa323" exitCode=0
Sep 11 18:26:15.952626 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.952485    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw"
Sep 11 18:26:15.952626 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.952506    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw" event={"ID":"c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe","Type":"ContainerDied","Data":"2c205cf83299a288c84d3743cdc4cd14c1cef09e7d6fa6fccb32401a621aa323"}
Sep 11 18:26:15.952626 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.952550    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw" event={"ID":"c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe","Type":"ContainerDied","Data":"cfe6cffed2c6d5fc7d069cdc2abc666232cfe12506a7e4f3e546cfbd9fa527d0"}
Sep 11 18:26:15.954159 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.954134    2576 generic.go:356] "Generic (PLEG): container finished" podID="d5ba777e-d6e1-4b50-a97c-3155a80a2400" containerID="9477f0feeec226cc6af56dec1d15a2b3bfa1c8530e6b0689fb8550ea4bcb67e6" exitCode=0
Sep 11 18:26:15.954271 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.954186    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb"
Sep 11 18:26:15.954271 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.954214    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb" event={"ID":"d5ba777e-d6e1-4b50-a97c-3155a80a2400","Type":"ContainerDied","Data":"9477f0feeec226cc6af56dec1d15a2b3bfa1c8530e6b0689fb8550ea4bcb67e6"}
Sep 11 18:26:15.954271 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.954264    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb" event={"ID":"d5ba777e-d6e1-4b50-a97c-3155a80a2400","Type":"ContainerDied","Data":"a8b574a62aa92eef0260209b400a9d84936d86791f74a7e1cc4668d44038702e"}
Sep 11 18:26:15.955726 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.955705    2576 generic.go:356] "Generic (PLEG): container finished" podID="2e144939-11fa-4ba0-b74c-900e823665f9" containerID="0ae1d92443dea91d65599bef8e43d05b5b6a3f6ead595869b37b4bbb29d7ebaf" exitCode=0
Sep 11 18:26:15.955822 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.955752    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52"
Sep 11 18:26:15.955822 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.955787    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52" event={"ID":"2e144939-11fa-4ba0-b74c-900e823665f9","Type":"ContainerDied","Data":"0ae1d92443dea91d65599bef8e43d05b5b6a3f6ead595869b37b4bbb29d7ebaf"}
Sep 11 18:26:15.955941 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.955852    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52" event={"ID":"2e144939-11fa-4ba0-b74c-900e823665f9","Type":"ContainerDied","Data":"1951fe26640116fd44c4b0e273a03b36a5f68770755b0cc6440f401a1e9ce95f"}
Sep 11 18:26:15.974010 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.973991    2576 scope.go:117] "RemoveContainer" containerID="1f6eb0fdbaf18e327f3e34cff226d415d22b3442865f60d98dd007d7ea910a21"
Sep 11 18:26:15.974331 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:26:15.974312    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f6eb0fdbaf18e327f3e34cff226d415d22b3442865f60d98dd007d7ea910a21\": container with ID starting with 1f6eb0fdbaf18e327f3e34cff226d415d22b3442865f60d98dd007d7ea910a21 not found: ID does not exist" containerID="1f6eb0fdbaf18e327f3e34cff226d415d22b3442865f60d98dd007d7ea910a21"
Sep 11 18:26:15.974392 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.974339    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f6eb0fdbaf18e327f3e34cff226d415d22b3442865f60d98dd007d7ea910a21"} err="failed to get container status \"1f6eb0fdbaf18e327f3e34cff226d415d22b3442865f60d98dd007d7ea910a21\": rpc error: code = NotFound desc = could not find container \"1f6eb0fdbaf18e327f3e34cff226d415d22b3442865f60d98dd007d7ea910a21\": container with ID starting with 1f6eb0fdbaf18e327f3e34cff226d415d22b3442865f60d98dd007d7ea910a21 not found: ID does not exist"
Sep 11 18:26:15.974392 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.974356    2576 scope.go:117] "RemoveContainer" containerID="2c205cf83299a288c84d3743cdc4cd14c1cef09e7d6fa6fccb32401a621aa323"
Sep 11 18:26:15.985005 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.984982    2576 scope.go:117] "RemoveContainer" containerID="2c205cf83299a288c84d3743cdc4cd14c1cef09e7d6fa6fccb32401a621aa323"
Sep 11 18:26:15.985639 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:26:15.985615    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c205cf83299a288c84d3743cdc4cd14c1cef09e7d6fa6fccb32401a621aa323\": container with ID starting with 2c205cf83299a288c84d3743cdc4cd14c1cef09e7d6fa6fccb32401a621aa323 not found: ID does not exist" containerID="2c205cf83299a288c84d3743cdc4cd14c1cef09e7d6fa6fccb32401a621aa323"
Sep 11 18:26:15.985730 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.985651    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c205cf83299a288c84d3743cdc4cd14c1cef09e7d6fa6fccb32401a621aa323"} err="failed to get container status \"2c205cf83299a288c84d3743cdc4cd14c1cef09e7d6fa6fccb32401a621aa323\": rpc error: code = NotFound desc = could not find container \"2c205cf83299a288c84d3743cdc4cd14c1cef09e7d6fa6fccb32401a621aa323\": container with ID starting with 2c205cf83299a288c84d3743cdc4cd14c1cef09e7d6fa6fccb32401a621aa323 not found: ID does not exist"
Sep 11 18:26:15.985730 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.985675    2576 scope.go:117] "RemoveContainer" containerID="9477f0feeec226cc6af56dec1d15a2b3bfa1c8530e6b0689fb8550ea4bcb67e6"
Sep 11 18:26:15.987720 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.987703    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc"]
Sep 11 18:26:15.989327 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.989303    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-89bb6c-68884df654-7r6bc"]
Sep 11 18:26:15.994223 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.994199    2576 scope.go:117] "RemoveContainer" containerID="9477f0feeec226cc6af56dec1d15a2b3bfa1c8530e6b0689fb8550ea4bcb67e6"
Sep 11 18:26:15.994476 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:26:15.994459    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9477f0feeec226cc6af56dec1d15a2b3bfa1c8530e6b0689fb8550ea4bcb67e6\": container with ID starting with 9477f0feeec226cc6af56dec1d15a2b3bfa1c8530e6b0689fb8550ea4bcb67e6 not found: ID does not exist" containerID="9477f0feeec226cc6af56dec1d15a2b3bfa1c8530e6b0689fb8550ea4bcb67e6"
Sep 11 18:26:15.994550 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.994487    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9477f0feeec226cc6af56dec1d15a2b3bfa1c8530e6b0689fb8550ea4bcb67e6"} err="failed to get container status \"9477f0feeec226cc6af56dec1d15a2b3bfa1c8530e6b0689fb8550ea4bcb67e6\": rpc error: code = NotFound desc = could not find container \"9477f0feeec226cc6af56dec1d15a2b3bfa1c8530e6b0689fb8550ea4bcb67e6\": container with ID starting with 9477f0feeec226cc6af56dec1d15a2b3bfa1c8530e6b0689fb8550ea4bcb67e6 not found: ID does not exist"
Sep 11 18:26:15.994550 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.994508    2576 scope.go:117] "RemoveContainer" containerID="0ae1d92443dea91d65599bef8e43d05b5b6a3f6ead595869b37b4bbb29d7ebaf"
Sep 11 18:26:15.998093 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:15.998070    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52"]
Sep 11 18:26:16.002136 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:16.002117    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-23f49f11-8656bfd986-4jt52"]
Sep 11 18:26:16.002299 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:16.002281    2576 scope.go:117] "RemoveContainer" containerID="0ae1d92443dea91d65599bef8e43d05b5b6a3f6ead595869b37b4bbb29d7ebaf"
Sep 11 18:26:16.002563 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:26:16.002546    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae1d92443dea91d65599bef8e43d05b5b6a3f6ead595869b37b4bbb29d7ebaf\": container with ID starting with 0ae1d92443dea91d65599bef8e43d05b5b6a3f6ead595869b37b4bbb29d7ebaf not found: ID does not exist" containerID="0ae1d92443dea91d65599bef8e43d05b5b6a3f6ead595869b37b4bbb29d7ebaf"
Sep 11 18:26:16.002647 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:16.002565    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae1d92443dea91d65599bef8e43d05b5b6a3f6ead595869b37b4bbb29d7ebaf"} err="failed to get container status \"0ae1d92443dea91d65599bef8e43d05b5b6a3f6ead595869b37b4bbb29d7ebaf\": rpc error: code = NotFound desc = could not find container \"0ae1d92443dea91d65599bef8e43d05b5b6a3f6ead595869b37b4bbb29d7ebaf\": container with ID starting with 0ae1d92443dea91d65599bef8e43d05b5b6a3f6ead595869b37b4bbb29d7ebaf not found: ID does not exist"
Sep 11 18:26:16.011860 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:16.011839    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw"]
Sep 11 18:26:16.017351 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:16.017332    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-23f49f11-b69749d7c-wnznw"]
Sep 11 18:26:16.028617 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:16.028593    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb"]
Sep 11 18:26:16.030445 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:16.030422    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-89bb6c-b765bbff7-q77gb"]
Sep 11 18:26:17.910594 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:17.910561    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e144939-11fa-4ba0-b74c-900e823665f9" path="/var/lib/kubelet/pods/2e144939-11fa-4ba0-b74c-900e823665f9/volumes"
Sep 11 18:26:17.911052 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:17.911036    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe" path="/var/lib/kubelet/pods/c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe/volumes"
Sep 11 18:26:17.911356 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:17.911344    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb2a3574-ea6e-4b22-9aa8-f103e36a47c0" path="/var/lib/kubelet/pods/cb2a3574-ea6e-4b22-9aa8-f103e36a47c0/volumes"
Sep 11 18:26:17.911673 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:17.911660    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5ba777e-d6e1-4b50-a97c-3155a80a2400" path="/var/lib/kubelet/pods/d5ba777e-d6e1-4b50-a97c-3155a80a2400/volumes"
Sep 11 18:26:32.205030 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:32.204952    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:26:34.552502 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.552472    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm"]
Sep 11 18:26:34.553034 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.553013    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e144939-11fa-4ba0-b74c-900e823665f9" containerName="payload-pre-processing"
Sep 11 18:26:34.553112 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.553038    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e144939-11fa-4ba0-b74c-900e823665f9" containerName="payload-pre-processing"
Sep 11 18:26:34.553112 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.553062    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5ba777e-d6e1-4b50-a97c-3155a80a2400" containerName="payload-processing"
Sep 11 18:26:34.553112 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.553071    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5ba777e-d6e1-4b50-a97c-3155a80a2400" containerName="payload-processing"
Sep 11 18:26:34.553112 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.553084    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb2a3574-ea6e-4b22-9aa8-f103e36a47c0" containerName="payload-pre-processing"
Sep 11 18:26:34.553112 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.553093    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb2a3574-ea6e-4b22-9aa8-f103e36a47c0" containerName="payload-pre-processing"
Sep 11 18:26:34.553112 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.553101    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe" containerName="payload-processing"
Sep 11 18:26:34.553112 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.553109    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe" containerName="payload-processing"
Sep 11 18:26:34.553439 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.553216    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e144939-11fa-4ba0-b74c-900e823665f9" containerName="payload-pre-processing"
Sep 11 18:26:34.553439 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.553227    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb2a3574-ea6e-4b22-9aa8-f103e36a47c0" containerName="payload-pre-processing"
Sep 11 18:26:34.553439 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.553239    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3e04c71-3e09-4ef7-8a7e-6bfe54e05ffe" containerName="payload-processing"
Sep 11 18:26:34.553439 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.553249    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5ba777e-d6e1-4b50-a97c-3155a80a2400" containerName="payload-processing"
Sep 11 18:26:34.556401 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.556382    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm"
Sep 11 18:26:34.558962 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.558929    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-386ac7\""
Sep 11 18:26:34.559074 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.559042    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-386ac7-dockercfg-prspc\""
Sep 11 18:26:34.567982 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.567952    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm"]
Sep 11 18:26:34.578664 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.578639    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf"]
Sep 11 18:26:34.582105 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.582083    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf"
Sep 11 18:26:34.593028 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.593001    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf"]
Sep 11 18:26:34.736170 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.736134    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr5j6\" (UniqueName: \"kubernetes.io/projected/e887951f-755c-419d-b218-10b9ac22c5c9-kube-api-access-vr5j6\") pod \"payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf\" (UID: \"e887951f-755c-419d-b218-10b9ac22c5c9\") " pod="openshift-ingress/payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf"
Sep 11 18:26:34.736346 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.736245    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztrpx\" (UniqueName: \"kubernetes.io/projected/883dbbe3-c42f-47f6-8f8a-6e2a72739ca1-kube-api-access-ztrpx\") pod \"payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm\" (UID: \"883dbbe3-c42f-47f6-8f8a-6e2a72739ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm"
Sep 11 18:26:34.736346 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.736274    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/883dbbe3-c42f-47f6-8f8a-6e2a72739ca1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm\" (UID: \"883dbbe3-c42f-47f6-8f8a-6e2a72739ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm"
Sep 11 18:26:34.736346 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.736320    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e887951f-755c-419d-b218-10b9ac22c5c9-plugins-config-volume\") pod \"payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf\" (UID: \"e887951f-755c-419d-b218-10b9ac22c5c9\") " pod="openshift-ingress/payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf"
Sep 11 18:26:34.837768 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.837664    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/883dbbe3-c42f-47f6-8f8a-6e2a72739ca1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm\" (UID: \"883dbbe3-c42f-47f6-8f8a-6e2a72739ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm"
Sep 11 18:26:34.837768 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.837714    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e887951f-755c-419d-b218-10b9ac22c5c9-plugins-config-volume\") pod \"payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf\" (UID: \"e887951f-755c-419d-b218-10b9ac22c5c9\") " pod="openshift-ingress/payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf"
Sep 11 18:26:34.837999 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.837844    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr5j6\" (UniqueName: \"kubernetes.io/projected/e887951f-755c-419d-b218-10b9ac22c5c9-kube-api-access-vr5j6\") pod \"payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf\" (UID: \"e887951f-755c-419d-b218-10b9ac22c5c9\") " pod="openshift-ingress/payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf"
Sep 11 18:26:34.837999 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.837961    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztrpx\" (UniqueName: \"kubernetes.io/projected/883dbbe3-c42f-47f6-8f8a-6e2a72739ca1-kube-api-access-ztrpx\") pod \"payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm\" (UID: \"883dbbe3-c42f-47f6-8f8a-6e2a72739ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm"
Sep 11 18:26:34.838525 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.838501    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e887951f-755c-419d-b218-10b9ac22c5c9-plugins-config-volume\") pod \"payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf\" (UID: \"e887951f-755c-419d-b218-10b9ac22c5c9\") " pod="openshift-ingress/payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf"
Sep 11 18:26:34.838672 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.838567    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/883dbbe3-c42f-47f6-8f8a-6e2a72739ca1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm\" (UID: \"883dbbe3-c42f-47f6-8f8a-6e2a72739ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm"
Sep 11 18:26:34.846445 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.846423    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr5j6\" (UniqueName: \"kubernetes.io/projected/e887951f-755c-419d-b218-10b9ac22c5c9-kube-api-access-vr5j6\") pod \"payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf\" (UID: \"e887951f-755c-419d-b218-10b9ac22c5c9\") " pod="openshift-ingress/payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf"
Sep 11 18:26:34.846670 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.846653    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztrpx\" (UniqueName: \"kubernetes.io/projected/883dbbe3-c42f-47f6-8f8a-6e2a72739ca1-kube-api-access-ztrpx\") pod \"payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm\" (UID: \"883dbbe3-c42f-47f6-8f8a-6e2a72739ca1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm"
Sep 11 18:26:34.866543 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.866510    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm"
Sep 11 18:26:34.892490 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:34.892458    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf"
Sep 11 18:26:35.139735 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:35.139706    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm"]
Sep 11 18:26:35.142208 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:26:35.142179    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883dbbe3_c42f_47f6_8f8a_6e2a72739ca1.slice/crio-b4bcbc2c52cb4d6abe14d496ffce6ab8a5602eb269cf787d61bdc3a65387d862 WatchSource:0}: Error finding container b4bcbc2c52cb4d6abe14d496ffce6ab8a5602eb269cf787d61bdc3a65387d862: Status 404 returned error can't find the container with id b4bcbc2c52cb4d6abe14d496ffce6ab8a5602eb269cf787d61bdc3a65387d862
Sep 11 18:26:35.174497 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:35.174475    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf"]
Sep 11 18:26:35.176840 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:26:35.176811    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode887951f_755c_419d_b218_10b9ac22c5c9.slice/crio-edb9f4297a82cc5943925a7a5f253f4d5ebe561f81222fa46fc9f39bc3aca4a9 WatchSource:0}: Error finding container edb9f4297a82cc5943925a7a5f253f4d5ebe561f81222fa46fc9f39bc3aca4a9: Status 404 returned error can't find the container with id edb9f4297a82cc5943925a7a5f253f4d5ebe561f81222fa46fc9f39bc3aca4a9
Sep 11 18:26:36.051148 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:36.051109    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf" event={"ID":"e887951f-755c-419d-b218-10b9ac22c5c9","Type":"ContainerStarted","Data":"1ca0290f15d71e382b47bdd004e6e2107c89b4b4c1ee6534f9d085950f5aebea"}
Sep 11 18:26:36.051690 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:36.051172    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf" event={"ID":"e887951f-755c-419d-b218-10b9ac22c5c9","Type":"ContainerStarted","Data":"edb9f4297a82cc5943925a7a5f253f4d5ebe561f81222fa46fc9f39bc3aca4a9"}
Sep 11 18:26:36.051690 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:36.051202    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf"
Sep 11 18:26:36.052827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:36.052797    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm" event={"ID":"883dbbe3-c42f-47f6-8f8a-6e2a72739ca1","Type":"ContainerStarted","Data":"fb0c3f3ca7b17851e8b5447970555cc960a637ebba7445e027e09a13128c4a2a"}
Sep 11 18:26:36.052931 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:36.052847    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm" event={"ID":"883dbbe3-c42f-47f6-8f8a-6e2a72739ca1","Type":"ContainerStarted","Data":"b4bcbc2c52cb4d6abe14d496ffce6ab8a5602eb269cf787d61bdc3a65387d862"}
Sep 11 18:26:36.052931 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:36.052875    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm"
Sep 11 18:26:36.069407 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:36.069355    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf" podStartSLOduration=2.069339087 podStartE2EDuration="2.069339087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:26:36.067995475 +0000 UTC m=+994.669356634" watchObservedRunningTime="2026-09-11 18:26:36.069339087 +0000 UTC m=+994.670700313"
Sep 11 18:26:47.060803 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:47.060768    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm"
Sep 11 18:26:47.061245 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:47.061163    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf"
Sep 11 18:26:47.081396 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:47.081345    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm" podStartSLOduration=13.081329853 podStartE2EDuration="13.081329853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:26:36.09032052 +0000 UTC m=+994.691681683" watchObservedRunningTime="2026-09-11 18:26:47.081329853 +0000 UTC m=+1005.682691015"
Sep 11 18:26:48.514418 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:26:48.514384    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:27:02.900944 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:02.900905    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:27:07.682069 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:07.682032    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"]
Sep 11 18:27:07.687183 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:07.687159    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:27:07.689595 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:07.689558    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-553596\"/\"premium-w4-553596-kserve-self-signed-certs\""
Sep 11 18:27:07.694960 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:07.694934    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"]
Sep 11 18:27:07.819739 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:07.819695    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-model-cache\") pod \"premium-w4-553596-kserve-65c5f4f79-97nzr\" (UID: \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\") " pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:27:07.819739 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:07.819743    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-dshm\") pod \"premium-w4-553596-kserve-65c5f4f79-97nzr\" (UID: \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\") " pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:27:07.819966 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:07.819830    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-tls-certs\") pod \"premium-w4-553596-kserve-65c5f4f79-97nzr\" (UID: \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\") " pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:27:07.819966 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:07.819875    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-home\") pod \"premium-w4-553596-kserve-65c5f4f79-97nzr\" (UID: \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\") " pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:27:07.819966 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:07.819892    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-tmp-dir\") pod \"premium-w4-553596-kserve-65c5f4f79-97nzr\" (UID: \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\") " pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:27:07.819966 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:07.819915    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwx7s\" (UniqueName: \"kubernetes.io/projected/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-kube-api-access-fwx7s\") pod \"premium-w4-553596-kserve-65c5f4f79-97nzr\" (UID: \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\") " pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:27:07.920888 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:07.920848    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-model-cache\") pod \"premium-w4-553596-kserve-65c5f4f79-97nzr\" (UID: \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\") " pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:27:07.920888 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:07.920898    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-dshm\") pod \"premium-w4-553596-kserve-65c5f4f79-97nzr\" (UID: \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\") " pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:27:07.921158 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:07.920934    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-tls-certs\") pod \"premium-w4-553596-kserve-65c5f4f79-97nzr\" (UID: \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\") " pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:27:07.921158 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:07.920966    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-home\") pod \"premium-w4-553596-kserve-65c5f4f79-97nzr\" (UID: \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\") " pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:27:07.921158 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:07.920983    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-tmp-dir\") pod \"premium-w4-553596-kserve-65c5f4f79-97nzr\" (UID: \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\") " pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:27:07.921158 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:07.920999    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwx7s\" (UniqueName: \"kubernetes.io/projected/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-kube-api-access-fwx7s\") pod \"premium-w4-553596-kserve-65c5f4f79-97nzr\" (UID: \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\") " pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:27:07.921369 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:07.921311    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-model-cache\") pod \"premium-w4-553596-kserve-65c5f4f79-97nzr\" (UID: \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\") " pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:27:07.921369 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:07.921348    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-tmp-dir\") pod \"premium-w4-553596-kserve-65c5f4f79-97nzr\" (UID: \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\") " pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:27:07.921474 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:07.921420    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-home\") pod \"premium-w4-553596-kserve-65c5f4f79-97nzr\" (UID: \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\") " pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:27:07.923324 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:07.923301    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-dshm\") pod \"premium-w4-553596-kserve-65c5f4f79-97nzr\" (UID: \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\") " pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:27:07.923453 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:07.923435    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-tls-certs\") pod \"premium-w4-553596-kserve-65c5f4f79-97nzr\" (UID: \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\") " pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:27:07.929331 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:07.929306    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwx7s\" (UniqueName: \"kubernetes.io/projected/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-kube-api-access-fwx7s\") pod \"premium-w4-553596-kserve-65c5f4f79-97nzr\" (UID: \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\") " pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:27:07.998468 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:07.998429    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:27:08.268965 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:08.268882    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"]
Sep 11 18:27:08.272132 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:27:08.272092    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd912ceb5_bbf0_44d3_8ea4_0b29099464cc.slice/crio-00e2529f2ae2d3d19362545dc736a21856fd72e033975c334045db05beff1bab WatchSource:0}: Error finding container 00e2529f2ae2d3d19362545dc736a21856fd72e033975c334045db05beff1bab: Status 404 returned error can't find the container with id 00e2529f2ae2d3d19362545dc736a21856fd72e033975c334045db05beff1bab
Sep 11 18:27:09.217129 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:09.217090    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr" event={"ID":"d912ceb5-bbf0-44d3-8ea4-0b29099464cc","Type":"ContainerStarted","Data":"0fdff7c16d3d26a2572564cf3718497b7ffb716573c964b74bf2d1ee5aad629d"}
Sep 11 18:27:09.217129 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:09.217136    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr" event={"ID":"d912ceb5-bbf0-44d3-8ea4-0b29099464cc","Type":"ContainerStarted","Data":"00e2529f2ae2d3d19362545dc736a21856fd72e033975c334045db05beff1bab"}
Sep 11 18:27:09.234873 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:09.234820    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr" podStartSLOduration=2.234804089 podStartE2EDuration="2.234804089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:09.233921825 +0000 UTC m=+1027.835283002" watchObservedRunningTime="2026-09-11 18:27:09.234804089 +0000 UTC m=+1027.836165251"
Sep 11 18:27:16.109962 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:16.109924    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:27:17.999327 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:17.999289    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:27:17.999327 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:17.999338    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:27:18.012256 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:18.012228    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:27:18.270618 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:18.270504    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:27:32.702046 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:32.702008    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:27:52.374206 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:52.374169    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf"]
Sep 11 18:27:52.374725 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:52.374494    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf" podUID="e887951f-755c-419d-b218-10b9ac22c5c9" containerName="payload-processing" containerID="cri-o://1ca0290f15d71e382b47bdd004e6e2107c89b4b4c1ee6534f9d085950f5aebea" gracePeriod=30
Sep 11 18:27:52.382080 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:52.381781    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm"]
Sep 11 18:27:52.382204 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:52.382043    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm" podUID="883dbbe3-c42f-47f6-8f8a-6e2a72739ca1" containerName="payload-pre-processing" containerID="cri-o://fb0c3f3ca7b17851e8b5447970555cc960a637ebba7445e027e09a13128c4a2a" gracePeriod=30
Sep 11 18:27:52.655568 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:52.655536    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf"
Sep 11 18:27:52.658726 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:52.658700    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm"
Sep 11 18:27:52.739371 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:52.739333    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/883dbbe3-c42f-47f6-8f8a-6e2a72739ca1-plugins-config-volume\") pod \"883dbbe3-c42f-47f6-8f8a-6e2a72739ca1\" (UID: \"883dbbe3-c42f-47f6-8f8a-6e2a72739ca1\") "
Sep 11 18:27:52.739558 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:52.739387    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vr5j6\" (UniqueName: \"kubernetes.io/projected/e887951f-755c-419d-b218-10b9ac22c5c9-kube-api-access-vr5j6\") pod \"e887951f-755c-419d-b218-10b9ac22c5c9\" (UID: \"e887951f-755c-419d-b218-10b9ac22c5c9\") "
Sep 11 18:27:52.739558 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:52.739434    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztrpx\" (UniqueName: \"kubernetes.io/projected/883dbbe3-c42f-47f6-8f8a-6e2a72739ca1-kube-api-access-ztrpx\") pod \"883dbbe3-c42f-47f6-8f8a-6e2a72739ca1\" (UID: \"883dbbe3-c42f-47f6-8f8a-6e2a72739ca1\") "
Sep 11 18:27:52.739558 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:52.739489    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e887951f-755c-419d-b218-10b9ac22c5c9-plugins-config-volume\") pod \"e887951f-755c-419d-b218-10b9ac22c5c9\" (UID: \"e887951f-755c-419d-b218-10b9ac22c5c9\") "
Sep 11 18:27:52.739977 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:52.739929    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e887951f-755c-419d-b218-10b9ac22c5c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e887951f-755c-419d-b218-10b9ac22c5c9" (UID: "e887951f-755c-419d-b218-10b9ac22c5c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:27:52.739977 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:52.739950    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/883dbbe3-c42f-47f6-8f8a-6e2a72739ca1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "883dbbe3-c42f-47f6-8f8a-6e2a72739ca1" (UID: "883dbbe3-c42f-47f6-8f8a-6e2a72739ca1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:27:52.741798 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:52.741768    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/883dbbe3-c42f-47f6-8f8a-6e2a72739ca1-kube-api-access-ztrpx" (OuterVolumeSpecName: "kube-api-access-ztrpx") pod "883dbbe3-c42f-47f6-8f8a-6e2a72739ca1" (UID: "883dbbe3-c42f-47f6-8f8a-6e2a72739ca1"). InnerVolumeSpecName "kube-api-access-ztrpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:27:52.741896 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:52.741830    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e887951f-755c-419d-b218-10b9ac22c5c9-kube-api-access-vr5j6" (OuterVolumeSpecName: "kube-api-access-vr5j6") pod "e887951f-755c-419d-b218-10b9ac22c5c9" (UID: "e887951f-755c-419d-b218-10b9ac22c5c9"). InnerVolumeSpecName "kube-api-access-vr5j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:27:52.840451 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:52.840414    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e887951f-755c-419d-b218-10b9ac22c5c9-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:27:52.840451 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:52.840447    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/883dbbe3-c42f-47f6-8f8a-6e2a72739ca1-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:27:52.840451 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:52.840461    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr5j6\" (UniqueName: \"kubernetes.io/projected/e887951f-755c-419d-b218-10b9ac22c5c9-kube-api-access-vr5j6\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:27:52.840695 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:52.840470    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztrpx\" (UniqueName: \"kubernetes.io/projected/883dbbe3-c42f-47f6-8f8a-6e2a72739ca1-kube-api-access-ztrpx\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:27:53.427736 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:53.427692    2576 generic.go:356] "Generic (PLEG): container finished" podID="e887951f-755c-419d-b218-10b9ac22c5c9" containerID="1ca0290f15d71e382b47bdd004e6e2107c89b4b4c1ee6534f9d085950f5aebea" exitCode=0
Sep 11 18:27:53.428216 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:53.427755    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf"
Sep 11 18:27:53.428216 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:53.427784    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf" event={"ID":"e887951f-755c-419d-b218-10b9ac22c5c9","Type":"ContainerDied","Data":"1ca0290f15d71e382b47bdd004e6e2107c89b4b4c1ee6534f9d085950f5aebea"}
Sep 11 18:27:53.428216 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:53.427896    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf" event={"ID":"e887951f-755c-419d-b218-10b9ac22c5c9","Type":"ContainerDied","Data":"edb9f4297a82cc5943925a7a5f253f4d5ebe561f81222fa46fc9f39bc3aca4a9"}
Sep 11 18:27:53.428216 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:53.427932    2576 scope.go:117] "RemoveContainer" containerID="1ca0290f15d71e382b47bdd004e6e2107c89b4b4c1ee6534f9d085950f5aebea"
Sep 11 18:27:53.429488 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:53.429334    2576 generic.go:356] "Generic (PLEG): container finished" podID="883dbbe3-c42f-47f6-8f8a-6e2a72739ca1" containerID="fb0c3f3ca7b17851e8b5447970555cc960a637ebba7445e027e09a13128c4a2a" exitCode=0
Sep 11 18:27:53.429488 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:53.429438    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm" event={"ID":"883dbbe3-c42f-47f6-8f8a-6e2a72739ca1","Type":"ContainerDied","Data":"fb0c3f3ca7b17851e8b5447970555cc960a637ebba7445e027e09a13128c4a2a"}
Sep 11 18:27:53.429488 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:53.429464    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm"
Sep 11 18:27:53.429658 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:53.429503    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm" event={"ID":"883dbbe3-c42f-47f6-8f8a-6e2a72739ca1","Type":"ContainerDied","Data":"b4bcbc2c52cb4d6abe14d496ffce6ab8a5602eb269cf787d61bdc3a65387d862"}
Sep 11 18:27:53.438101 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:53.438077    2576 scope.go:117] "RemoveContainer" containerID="1ca0290f15d71e382b47bdd004e6e2107c89b4b4c1ee6534f9d085950f5aebea"
Sep 11 18:27:53.438419 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:27:53.438393    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ca0290f15d71e382b47bdd004e6e2107c89b4b4c1ee6534f9d085950f5aebea\": container with ID starting with 1ca0290f15d71e382b47bdd004e6e2107c89b4b4c1ee6534f9d085950f5aebea not found: ID does not exist" containerID="1ca0290f15d71e382b47bdd004e6e2107c89b4b4c1ee6534f9d085950f5aebea"
Sep 11 18:27:53.438515 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:53.438429    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca0290f15d71e382b47bdd004e6e2107c89b4b4c1ee6534f9d085950f5aebea"} err="failed to get container status \"1ca0290f15d71e382b47bdd004e6e2107c89b4b4c1ee6534f9d085950f5aebea\": rpc error: code = NotFound desc = could not find container \"1ca0290f15d71e382b47bdd004e6e2107c89b4b4c1ee6534f9d085950f5aebea\": container with ID starting with 1ca0290f15d71e382b47bdd004e6e2107c89b4b4c1ee6534f9d085950f5aebea not found: ID does not exist"
Sep 11 18:27:53.438515 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:53.438451    2576 scope.go:117] "RemoveContainer" containerID="fb0c3f3ca7b17851e8b5447970555cc960a637ebba7445e027e09a13128c4a2a"
Sep 11 18:27:53.446676 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:53.446659    2576 scope.go:117] "RemoveContainer" containerID="fb0c3f3ca7b17851e8b5447970555cc960a637ebba7445e027e09a13128c4a2a"
Sep 11 18:27:53.446946 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:27:53.446926    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb0c3f3ca7b17851e8b5447970555cc960a637ebba7445e027e09a13128c4a2a\": container with ID starting with fb0c3f3ca7b17851e8b5447970555cc960a637ebba7445e027e09a13128c4a2a not found: ID does not exist" containerID="fb0c3f3ca7b17851e8b5447970555cc960a637ebba7445e027e09a13128c4a2a"
Sep 11 18:27:53.447026 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:53.446952    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb0c3f3ca7b17851e8b5447970555cc960a637ebba7445e027e09a13128c4a2a"} err="failed to get container status \"fb0c3f3ca7b17851e8b5447970555cc960a637ebba7445e027e09a13128c4a2a\": rpc error: code = NotFound desc = could not find container \"fb0c3f3ca7b17851e8b5447970555cc960a637ebba7445e027e09a13128c4a2a\": container with ID starting with fb0c3f3ca7b17851e8b5447970555cc960a637ebba7445e027e09a13128c4a2a not found: ID does not exist"
Sep 11 18:27:53.451219 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:53.451199    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf"]
Sep 11 18:27:53.457159 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:53.457141    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-386ac7-7dbdc767b9-zw7xf"]
Sep 11 18:27:53.466489 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:53.466465    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm"]
Sep 11 18:27:53.472412 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:53.472390    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-386ac7-6d9f54ff4b-sx4lm"]
Sep 11 18:27:53.911009 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:53.910975    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="883dbbe3-c42f-47f6-8f8a-6e2a72739ca1" path="/var/lib/kubelet/pods/883dbbe3-c42f-47f6-8f8a-6e2a72739ca1/volumes"
Sep 11 18:27:53.911352 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:27:53.911339    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e887951f-755c-419d-b218-10b9ac22c5c9" path="/var/lib/kubelet/pods/e887951f-755c-419d-b218-10b9ac22c5c9/volumes"
Sep 11 18:28:00.304086 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:00.304049    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:28:04.721395 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:04.721353    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb"]
Sep 11 18:28:04.722157 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:04.722130    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e887951f-755c-419d-b218-10b9ac22c5c9" containerName="payload-processing"
Sep 11 18:28:04.722157 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:04.722160    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e887951f-755c-419d-b218-10b9ac22c5c9" containerName="payload-processing"
Sep 11 18:28:04.722298 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:04.722187    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="883dbbe3-c42f-47f6-8f8a-6e2a72739ca1" containerName="payload-pre-processing"
Sep 11 18:28:04.722298 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:04.722195    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="883dbbe3-c42f-47f6-8f8a-6e2a72739ca1" containerName="payload-pre-processing"
Sep 11 18:28:04.722388 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:04.722362    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e887951f-755c-419d-b218-10b9ac22c5c9" containerName="payload-processing"
Sep 11 18:28:04.722388 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:04.722383    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="883dbbe3-c42f-47f6-8f8a-6e2a72739ca1" containerName="payload-pre-processing"
Sep 11 18:28:04.727442 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:04.727419    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb"
Sep 11 18:28:04.729612 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:04.729567    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-07f2626b\""
Sep 11 18:28:04.729728 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:04.729711    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-07f2626b-dockercfg-xdxcj\""
Sep 11 18:28:04.733552 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:04.733522    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb"]
Sep 11 18:28:04.759515 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:04.759487    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc"]
Sep 11 18:28:04.762803 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:04.762786    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc"
Sep 11 18:28:04.772067 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:04.772044    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc"]
Sep 11 18:28:04.851080 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:04.851047    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b85c0a1-a06e-4221-919f-5005b91984ee-plugins-config-volume\") pod \"payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb\" (UID: \"8b85c0a1-a06e-4221-919f-5005b91984ee\") " pod="openshift-ingress/payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb"
Sep 11 18:28:04.851258 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:04.851084    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7hh6\" (UniqueName: \"kubernetes.io/projected/8b85c0a1-a06e-4221-919f-5005b91984ee-kube-api-access-p7hh6\") pod \"payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb\" (UID: \"8b85c0a1-a06e-4221-919f-5005b91984ee\") " pod="openshift-ingress/payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb"
Sep 11 18:28:04.952723 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:04.952677    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b85c0a1-a06e-4221-919f-5005b91984ee-plugins-config-volume\") pod \"payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb\" (UID: \"8b85c0a1-a06e-4221-919f-5005b91984ee\") " pod="openshift-ingress/payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb"
Sep 11 18:28:04.952723 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:04.952725    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7hh6\" (UniqueName: \"kubernetes.io/projected/8b85c0a1-a06e-4221-919f-5005b91984ee-kube-api-access-p7hh6\") pod \"payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb\" (UID: \"8b85c0a1-a06e-4221-919f-5005b91984ee\") " pod="openshift-ingress/payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb"
Sep 11 18:28:04.952969 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:04.952787    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkhvp\" (UniqueName: \"kubernetes.io/projected/9a66df93-d727-409a-8c4c-723014d91309-kube-api-access-hkhvp\") pod \"payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc\" (UID: \"9a66df93-d727-409a-8c4c-723014d91309\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc"
Sep 11 18:28:04.952969 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:04.952810    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a66df93-d727-409a-8c4c-723014d91309-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc\" (UID: \"9a66df93-d727-409a-8c4c-723014d91309\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc"
Sep 11 18:28:04.953431 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:04.953411    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b85c0a1-a06e-4221-919f-5005b91984ee-plugins-config-volume\") pod \"payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb\" (UID: \"8b85c0a1-a06e-4221-919f-5005b91984ee\") " pod="openshift-ingress/payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb"
Sep 11 18:28:04.961123 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:04.961099    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7hh6\" (UniqueName: \"kubernetes.io/projected/8b85c0a1-a06e-4221-919f-5005b91984ee-kube-api-access-p7hh6\") pod \"payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb\" (UID: \"8b85c0a1-a06e-4221-919f-5005b91984ee\") " pod="openshift-ingress/payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb"
Sep 11 18:28:05.039123 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:05.039024    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb"
Sep 11 18:28:05.053754 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:05.053724    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkhvp\" (UniqueName: \"kubernetes.io/projected/9a66df93-d727-409a-8c4c-723014d91309-kube-api-access-hkhvp\") pod \"payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc\" (UID: \"9a66df93-d727-409a-8c4c-723014d91309\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc"
Sep 11 18:28:05.053896 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:05.053770    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a66df93-d727-409a-8c4c-723014d91309-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc\" (UID: \"9a66df93-d727-409a-8c4c-723014d91309\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc"
Sep 11 18:28:05.054435 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:05.054415    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a66df93-d727-409a-8c4c-723014d91309-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc\" (UID: \"9a66df93-d727-409a-8c4c-723014d91309\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc"
Sep 11 18:28:05.063211 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:05.063183    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkhvp\" (UniqueName: \"kubernetes.io/projected/9a66df93-d727-409a-8c4c-723014d91309-kube-api-access-hkhvp\") pod \"payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc\" (UID: \"9a66df93-d727-409a-8c4c-723014d91309\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc"
Sep 11 18:28:05.074061 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:05.074033    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc"
Sep 11 18:28:05.318128 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:05.318097    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb"]
Sep 11 18:28:05.320632 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:28:05.320600    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b85c0a1_a06e_4221_919f_5005b91984ee.slice/crio-63d975ddfb3d6687b6cdf59bce99acbd7ce45efeeeb334233e819cc1f0d83bcd WatchSource:0}: Error finding container 63d975ddfb3d6687b6cdf59bce99acbd7ce45efeeeb334233e819cc1f0d83bcd: Status 404 returned error can't find the container with id 63d975ddfb3d6687b6cdf59bce99acbd7ce45efeeeb334233e819cc1f0d83bcd
Sep 11 18:28:05.362522 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:05.362497    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc"]
Sep 11 18:28:05.488792 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:05.488757    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc" event={"ID":"9a66df93-d727-409a-8c4c-723014d91309","Type":"ContainerStarted","Data":"8c5e5ff65cc869c5eaff74925a1aff3db6abff1b752479d4e3f6743e1ea2d1e3"}
Sep 11 18:28:05.488980 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:05.488820    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc" event={"ID":"9a66df93-d727-409a-8c4c-723014d91309","Type":"ContainerStarted","Data":"d9e899083a5888461c1e1b686a4b5def5ad28e4f991796900c8c39a85edf3a98"}
Sep 11 18:28:05.488980 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:05.488853    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc"
Sep 11 18:28:05.490458 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:05.490428    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb" event={"ID":"8b85c0a1-a06e-4221-919f-5005b91984ee","Type":"ContainerStarted","Data":"31d4a590892c904bcede1abf4b608e5abd42193945c2a556ae08be7684e30b1e"}
Sep 11 18:28:05.490613 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:05.490467    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb" event={"ID":"8b85c0a1-a06e-4221-919f-5005b91984ee","Type":"ContainerStarted","Data":"63d975ddfb3d6687b6cdf59bce99acbd7ce45efeeeb334233e819cc1f0d83bcd"}
Sep 11 18:28:05.490613 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:05.490533    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb"
Sep 11 18:28:05.506393 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:05.506347    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc" podStartSLOduration=1.506331759 podStartE2EDuration="1.506331759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:28:05.50445062 +0000 UTC m=+1084.105811782" watchObservedRunningTime="2026-09-11 18:28:05.506331759 +0000 UTC m=+1084.107692919"
Sep 11 18:28:05.523142 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:05.523092    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb" podStartSLOduration=1.52307712 podStartE2EDuration="1.52307712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:28:05.522224399 +0000 UTC m=+1084.123585562" watchObservedRunningTime="2026-09-11 18:28:05.52307712 +0000 UTC m=+1084.124438283"
Sep 11 18:28:16.497766 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:16.497728    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb"
Sep 11 18:28:16.498148 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:16.498032    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc"
Sep 11 18:28:34.217922 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:34.217887    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:28:48.753733 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:48.753700    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb"]
Sep 11 18:28:48.754134 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:48.753913    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb" podUID="8b85c0a1-a06e-4221-919f-5005b91984ee" containerName="payload-processing" containerID="cri-o://31d4a590892c904bcede1abf4b608e5abd42193945c2a556ae08be7684e30b1e" gracePeriod=30
Sep 11 18:28:48.764377 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:48.764342    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc"]
Sep 11 18:28:48.764642 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:48.764577    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc" podUID="9a66df93-d727-409a-8c4c-723014d91309" containerName="payload-pre-processing" containerID="cri-o://8c5e5ff65cc869c5eaff74925a1aff3db6abff1b752479d4e3f6743e1ea2d1e3" gracePeriod=30
Sep 11 18:28:49.042562 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.042533    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb"
Sep 11 18:28:49.045538 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.045520    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc"
Sep 11 18:28:49.145188 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.145160    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7hh6\" (UniqueName: \"kubernetes.io/projected/8b85c0a1-a06e-4221-919f-5005b91984ee-kube-api-access-p7hh6\") pod \"8b85c0a1-a06e-4221-919f-5005b91984ee\" (UID: \"8b85c0a1-a06e-4221-919f-5005b91984ee\") "
Sep 11 18:28:49.145349 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.145197    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a66df93-d727-409a-8c4c-723014d91309-plugins-config-volume\") pod \"9a66df93-d727-409a-8c4c-723014d91309\" (UID: \"9a66df93-d727-409a-8c4c-723014d91309\") "
Sep 11 18:28:49.145349 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.145305    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b85c0a1-a06e-4221-919f-5005b91984ee-plugins-config-volume\") pod \"8b85c0a1-a06e-4221-919f-5005b91984ee\" (UID: \"8b85c0a1-a06e-4221-919f-5005b91984ee\") "
Sep 11 18:28:49.145349 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.145323    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkhvp\" (UniqueName: \"kubernetes.io/projected/9a66df93-d727-409a-8c4c-723014d91309-kube-api-access-hkhvp\") pod \"9a66df93-d727-409a-8c4c-723014d91309\" (UID: \"9a66df93-d727-409a-8c4c-723014d91309\") "
Sep 11 18:28:49.145717 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.145689    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b85c0a1-a06e-4221-919f-5005b91984ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b85c0a1-a06e-4221-919f-5005b91984ee" (UID: "8b85c0a1-a06e-4221-919f-5005b91984ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:28:49.145717 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.145701    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a66df93-d727-409a-8c4c-723014d91309-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9a66df93-d727-409a-8c4c-723014d91309" (UID: "9a66df93-d727-409a-8c4c-723014d91309"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:28:49.147218 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.147195    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b85c0a1-a06e-4221-919f-5005b91984ee-kube-api-access-p7hh6" (OuterVolumeSpecName: "kube-api-access-p7hh6") pod "8b85c0a1-a06e-4221-919f-5005b91984ee" (UID: "8b85c0a1-a06e-4221-919f-5005b91984ee"). InnerVolumeSpecName "kube-api-access-p7hh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:28:49.147342 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.147326    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a66df93-d727-409a-8c4c-723014d91309-kube-api-access-hkhvp" (OuterVolumeSpecName: "kube-api-access-hkhvp") pod "9a66df93-d727-409a-8c4c-723014d91309" (UID: "9a66df93-d727-409a-8c4c-723014d91309"). InnerVolumeSpecName "kube-api-access-hkhvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:28:49.246073 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.246030    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b85c0a1-a06e-4221-919f-5005b91984ee-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:28:49.246073 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.246065    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkhvp\" (UniqueName: \"kubernetes.io/projected/9a66df93-d727-409a-8c4c-723014d91309-kube-api-access-hkhvp\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:28:49.246073 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.246076    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7hh6\" (UniqueName: \"kubernetes.io/projected/8b85c0a1-a06e-4221-919f-5005b91984ee-kube-api-access-p7hh6\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:28:49.246295 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.246085    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a66df93-d727-409a-8c4c-723014d91309-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:28:49.714973 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.714935    2576 generic.go:356] "Generic (PLEG): container finished" podID="8b85c0a1-a06e-4221-919f-5005b91984ee" containerID="31d4a590892c904bcede1abf4b608e5abd42193945c2a556ae08be7684e30b1e" exitCode=0
Sep 11 18:28:49.715151 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.714985    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb"
Sep 11 18:28:49.715151 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.715023    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb" event={"ID":"8b85c0a1-a06e-4221-919f-5005b91984ee","Type":"ContainerDied","Data":"31d4a590892c904bcede1abf4b608e5abd42193945c2a556ae08be7684e30b1e"}
Sep 11 18:28:49.715151 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.715082    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb" event={"ID":"8b85c0a1-a06e-4221-919f-5005b91984ee","Type":"ContainerDied","Data":"63d975ddfb3d6687b6cdf59bce99acbd7ce45efeeeb334233e819cc1f0d83bcd"}
Sep 11 18:28:49.715151 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.715106    2576 scope.go:117] "RemoveContainer" containerID="31d4a590892c904bcede1abf4b608e5abd42193945c2a556ae08be7684e30b1e"
Sep 11 18:28:49.716434 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.716411    2576 generic.go:356] "Generic (PLEG): container finished" podID="9a66df93-d727-409a-8c4c-723014d91309" containerID="8c5e5ff65cc869c5eaff74925a1aff3db6abff1b752479d4e3f6743e1ea2d1e3" exitCode=0
Sep 11 18:28:49.716518 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.716454    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc"
Sep 11 18:28:49.716518 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.716473    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc" event={"ID":"9a66df93-d727-409a-8c4c-723014d91309","Type":"ContainerDied","Data":"8c5e5ff65cc869c5eaff74925a1aff3db6abff1b752479d4e3f6743e1ea2d1e3"}
Sep 11 18:28:49.716618 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.716515    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc" event={"ID":"9a66df93-d727-409a-8c4c-723014d91309","Type":"ContainerDied","Data":"d9e899083a5888461c1e1b686a4b5def5ad28e4f991796900c8c39a85edf3a98"}
Sep 11 18:28:49.724850 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.724832    2576 scope.go:117] "RemoveContainer" containerID="31d4a590892c904bcede1abf4b608e5abd42193945c2a556ae08be7684e30b1e"
Sep 11 18:28:49.725143 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:28:49.725127    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31d4a590892c904bcede1abf4b608e5abd42193945c2a556ae08be7684e30b1e\": container with ID starting with 31d4a590892c904bcede1abf4b608e5abd42193945c2a556ae08be7684e30b1e not found: ID does not exist" containerID="31d4a590892c904bcede1abf4b608e5abd42193945c2a556ae08be7684e30b1e"
Sep 11 18:28:49.725223 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.725152    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31d4a590892c904bcede1abf4b608e5abd42193945c2a556ae08be7684e30b1e"} err="failed to get container status \"31d4a590892c904bcede1abf4b608e5abd42193945c2a556ae08be7684e30b1e\": rpc error: code = NotFound desc = could not find container \"31d4a590892c904bcede1abf4b608e5abd42193945c2a556ae08be7684e30b1e\": container with ID starting with 31d4a590892c904bcede1abf4b608e5abd42193945c2a556ae08be7684e30b1e not found: ID does not exist"
Sep 11 18:28:49.725223 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.725167    2576 scope.go:117] "RemoveContainer" containerID="8c5e5ff65cc869c5eaff74925a1aff3db6abff1b752479d4e3f6743e1ea2d1e3"
Sep 11 18:28:49.733370 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.733353    2576 scope.go:117] "RemoveContainer" containerID="8c5e5ff65cc869c5eaff74925a1aff3db6abff1b752479d4e3f6743e1ea2d1e3"
Sep 11 18:28:49.733671 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:28:49.733653    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c5e5ff65cc869c5eaff74925a1aff3db6abff1b752479d4e3f6743e1ea2d1e3\": container with ID starting with 8c5e5ff65cc869c5eaff74925a1aff3db6abff1b752479d4e3f6743e1ea2d1e3 not found: ID does not exist" containerID="8c5e5ff65cc869c5eaff74925a1aff3db6abff1b752479d4e3f6743e1ea2d1e3"
Sep 11 18:28:49.733740 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.733679    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c5e5ff65cc869c5eaff74925a1aff3db6abff1b752479d4e3f6743e1ea2d1e3"} err="failed to get container status \"8c5e5ff65cc869c5eaff74925a1aff3db6abff1b752479d4e3f6743e1ea2d1e3\": rpc error: code = NotFound desc = could not find container \"8c5e5ff65cc869c5eaff74925a1aff3db6abff1b752479d4e3f6743e1ea2d1e3\": container with ID starting with 8c5e5ff65cc869c5eaff74925a1aff3db6abff1b752479d4e3f6743e1ea2d1e3 not found: ID does not exist"
Sep 11 18:28:49.739503 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.739479    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb"]
Sep 11 18:28:49.745448 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.745425    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-07f2626b-557d6bc885-gqpsb"]
Sep 11 18:28:49.755504 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.755476    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc"]
Sep 11 18:28:49.762320 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.762298    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-07f2626b-564b5f4f9d-dkfsc"]
Sep 11 18:28:49.911040 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.911007    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b85c0a1-a06e-4221-919f-5005b91984ee" path="/var/lib/kubelet/pods/8b85c0a1-a06e-4221-919f-5005b91984ee/volumes"
Sep 11 18:28:49.911427 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:49.911413    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a66df93-d727-409a-8c4c-723014d91309" path="/var/lib/kubelet/pods/9a66df93-d727-409a-8c4c-723014d91309/volumes"
Sep 11 18:28:57.377625 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:28:57.377577    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:29:15.705116 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.705078    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"]
Sep 11 18:29:15.705626 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.705447    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a66df93-d727-409a-8c4c-723014d91309" containerName="payload-pre-processing"
Sep 11 18:29:15.705626 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.705460    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a66df93-d727-409a-8c4c-723014d91309" containerName="payload-pre-processing"
Sep 11 18:29:15.705626 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.705470    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b85c0a1-a06e-4221-919f-5005b91984ee" containerName="payload-processing"
Sep 11 18:29:15.705626 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.705477    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b85c0a1-a06e-4221-919f-5005b91984ee" containerName="payload-processing"
Sep 11 18:29:15.705626 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.705536    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a66df93-d727-409a-8c4c-723014d91309" containerName="payload-pre-processing"
Sep 11 18:29:15.705626 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.705549    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b85c0a1-a06e-4221-919f-5005b91984ee" containerName="payload-processing"
Sep 11 18:29:15.709879 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.709856    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:29:15.713238 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.713222    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d73031\"/\"kube-root-ca.crt\""
Sep 11 18:29:15.713298 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.713277    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d73031\"/\"default-dockercfg-n6mnk\""
Sep 11 18:29:15.713298 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.713220    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d73031\"/\"openshift-service-ca.crt\""
Sep 11 18:29:15.713386 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.713223    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d73031\"/\"distinct-w0-d73031-kserve-self-signed-certs\""
Sep 11 18:29:15.722300 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.722280    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"]
Sep 11 18:29:15.792058 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.792020    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c73362b7-f9eb-4789-8d84-062491a7dd45-model-cache\") pod \"distinct-w0-d73031-kserve-68ff5f68fc-4z24n\" (UID: \"c73362b7-f9eb-4789-8d84-062491a7dd45\") " pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:29:15.792058 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.792063    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c73362b7-f9eb-4789-8d84-062491a7dd45-dshm\") pod \"distinct-w0-d73031-kserve-68ff5f68fc-4z24n\" (UID: \"c73362b7-f9eb-4789-8d84-062491a7dd45\") " pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:29:15.792300 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.792098    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c73362b7-f9eb-4789-8d84-062491a7dd45-home\") pod \"distinct-w0-d73031-kserve-68ff5f68fc-4z24n\" (UID: \"c73362b7-f9eb-4789-8d84-062491a7dd45\") " pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:29:15.792300 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.792144    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k56lv\" (UniqueName: \"kubernetes.io/projected/c73362b7-f9eb-4789-8d84-062491a7dd45-kube-api-access-k56lv\") pod \"distinct-w0-d73031-kserve-68ff5f68fc-4z24n\" (UID: \"c73362b7-f9eb-4789-8d84-062491a7dd45\") " pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:29:15.792300 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.792185    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c73362b7-f9eb-4789-8d84-062491a7dd45-tls-certs\") pod \"distinct-w0-d73031-kserve-68ff5f68fc-4z24n\" (UID: \"c73362b7-f9eb-4789-8d84-062491a7dd45\") " pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:29:15.792300 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.792221    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c73362b7-f9eb-4789-8d84-062491a7dd45-tmp-dir\") pod \"distinct-w0-d73031-kserve-68ff5f68fc-4z24n\" (UID: \"c73362b7-f9eb-4789-8d84-062491a7dd45\") " pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:29:15.893487 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.893439    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c73362b7-f9eb-4789-8d84-062491a7dd45-model-cache\") pod \"distinct-w0-d73031-kserve-68ff5f68fc-4z24n\" (UID: \"c73362b7-f9eb-4789-8d84-062491a7dd45\") " pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:29:15.893487 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.893491    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c73362b7-f9eb-4789-8d84-062491a7dd45-dshm\") pod \"distinct-w0-d73031-kserve-68ff5f68fc-4z24n\" (UID: \"c73362b7-f9eb-4789-8d84-062491a7dd45\") " pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:29:15.893796 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.893534    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c73362b7-f9eb-4789-8d84-062491a7dd45-home\") pod \"distinct-w0-d73031-kserve-68ff5f68fc-4z24n\" (UID: \"c73362b7-f9eb-4789-8d84-062491a7dd45\") " pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:29:15.893796 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.893550    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k56lv\" (UniqueName: \"kubernetes.io/projected/c73362b7-f9eb-4789-8d84-062491a7dd45-kube-api-access-k56lv\") pod \"distinct-w0-d73031-kserve-68ff5f68fc-4z24n\" (UID: \"c73362b7-f9eb-4789-8d84-062491a7dd45\") " pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:29:15.893796 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.893569    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c73362b7-f9eb-4789-8d84-062491a7dd45-tls-certs\") pod \"distinct-w0-d73031-kserve-68ff5f68fc-4z24n\" (UID: \"c73362b7-f9eb-4789-8d84-062491a7dd45\") " pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:29:15.893796 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.893618    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c73362b7-f9eb-4789-8d84-062491a7dd45-tmp-dir\") pod \"distinct-w0-d73031-kserve-68ff5f68fc-4z24n\" (UID: \"c73362b7-f9eb-4789-8d84-062491a7dd45\") " pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:29:15.894025 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.893944    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c73362b7-f9eb-4789-8d84-062491a7dd45-model-cache\") pod \"distinct-w0-d73031-kserve-68ff5f68fc-4z24n\" (UID: \"c73362b7-f9eb-4789-8d84-062491a7dd45\") " pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:29:15.894025 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.893981    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c73362b7-f9eb-4789-8d84-062491a7dd45-home\") pod \"distinct-w0-d73031-kserve-68ff5f68fc-4z24n\" (UID: \"c73362b7-f9eb-4789-8d84-062491a7dd45\") " pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:29:15.894025 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.894010    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c73362b7-f9eb-4789-8d84-062491a7dd45-tmp-dir\") pod \"distinct-w0-d73031-kserve-68ff5f68fc-4z24n\" (UID: \"c73362b7-f9eb-4789-8d84-062491a7dd45\") " pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:29:15.895840 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.895812    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c73362b7-f9eb-4789-8d84-062491a7dd45-dshm\") pod \"distinct-w0-d73031-kserve-68ff5f68fc-4z24n\" (UID: \"c73362b7-f9eb-4789-8d84-062491a7dd45\") " pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:29:15.896081 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.896066    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c73362b7-f9eb-4789-8d84-062491a7dd45-tls-certs\") pod \"distinct-w0-d73031-kserve-68ff5f68fc-4z24n\" (UID: \"c73362b7-f9eb-4789-8d84-062491a7dd45\") " pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:29:15.901702 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:15.901682    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k56lv\" (UniqueName: \"kubernetes.io/projected/c73362b7-f9eb-4789-8d84-062491a7dd45-kube-api-access-k56lv\") pod \"distinct-w0-d73031-kserve-68ff5f68fc-4z24n\" (UID: \"c73362b7-f9eb-4789-8d84-062491a7dd45\") " pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:29:16.019036 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:16.018936    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:29:16.293865 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:16.293836    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"]
Sep 11 18:29:16.296633 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:29:16.296601    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc73362b7_f9eb_4789_8d84_062491a7dd45.slice/crio-d6942ac8d2c98383dc7593792534198536cb11dd1e076d8bfbc9fda08b917198 WatchSource:0}: Error finding container d6942ac8d2c98383dc7593792534198536cb11dd1e076d8bfbc9fda08b917198: Status 404 returned error can't find the container with id d6942ac8d2c98383dc7593792534198536cb11dd1e076d8bfbc9fda08b917198
Sep 11 18:29:16.848461 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:16.848427    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n" event={"ID":"c73362b7-f9eb-4789-8d84-062491a7dd45","Type":"ContainerStarted","Data":"d8e424845dea51271bf5f59eb74799855bc3e7483118b21dc876a414b27b6e10"}
Sep 11 18:29:16.848461 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:16.848468    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n" event={"ID":"c73362b7-f9eb-4789-8d84-062491a7dd45","Type":"ContainerStarted","Data":"d6942ac8d2c98383dc7593792534198536cb11dd1e076d8bfbc9fda08b917198"}
Sep 11 18:29:16.868362 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:16.868313    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n" podStartSLOduration=1.8682966859999999 podStartE2EDuration="1.868296686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:29:16.866379483 +0000 UTC m=+1155.467740643" watchObservedRunningTime="2026-09-11 18:29:16.868296686 +0000 UTC m=+1155.469657848"
Sep 11 18:29:17.133236 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:17.133154    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:29:17.903541 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:17.903510    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt"]
Sep 11 18:29:17.906945 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:17.906925    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt"
Sep 11 18:29:17.909976 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:17.909953    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-a761e9\""
Sep 11 18:29:17.910087 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:17.910039    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-a761e9-dockercfg-dxkj8\""
Sep 11 18:29:17.921531 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:17.921509    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt"]
Sep 11 18:29:17.935939 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:17.935910    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb"]
Sep 11 18:29:17.939223 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:17.939203    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb"
Sep 11 18:29:17.953037 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:17.953011    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb"]
Sep 11 18:29:18.015181 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:18.015141    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14a667c-e9e9-41ba-8bd7-105565a09431-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt\" (UID: \"e14a667c-e9e9-41ba-8bd7-105565a09431\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt"
Sep 11 18:29:18.015381 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:18.015201    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6546fdce-36a2-4a07-a06c-4786da61d190-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb\" (UID: \"6546fdce-36a2-4a07-a06c-4786da61d190\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb"
Sep 11 18:29:18.015381 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:18.015329    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl7cj\" (UniqueName: \"kubernetes.io/projected/e14a667c-e9e9-41ba-8bd7-105565a09431-kube-api-access-tl7cj\") pod \"payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt\" (UID: \"e14a667c-e9e9-41ba-8bd7-105565a09431\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt"
Sep 11 18:29:18.015507 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:18.015465    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zv8v\" (UniqueName: \"kubernetes.io/projected/6546fdce-36a2-4a07-a06c-4786da61d190-kube-api-access-9zv8v\") pod \"payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb\" (UID: \"6546fdce-36a2-4a07-a06c-4786da61d190\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb"
Sep 11 18:29:18.116051 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:18.116010    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zv8v\" (UniqueName: \"kubernetes.io/projected/6546fdce-36a2-4a07-a06c-4786da61d190-kube-api-access-9zv8v\") pod \"payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb\" (UID: \"6546fdce-36a2-4a07-a06c-4786da61d190\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb"
Sep 11 18:29:18.116256 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:18.116123    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14a667c-e9e9-41ba-8bd7-105565a09431-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt\" (UID: \"e14a667c-e9e9-41ba-8bd7-105565a09431\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt"
Sep 11 18:29:18.116256 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:18.116159    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6546fdce-36a2-4a07-a06c-4786da61d190-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb\" (UID: \"6546fdce-36a2-4a07-a06c-4786da61d190\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb"
Sep 11 18:29:18.116256 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:18.116187    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl7cj\" (UniqueName: \"kubernetes.io/projected/e14a667c-e9e9-41ba-8bd7-105565a09431-kube-api-access-tl7cj\") pod \"payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt\" (UID: \"e14a667c-e9e9-41ba-8bd7-105565a09431\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt"
Sep 11 18:29:18.116912 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:18.116894    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6546fdce-36a2-4a07-a06c-4786da61d190-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb\" (UID: \"6546fdce-36a2-4a07-a06c-4786da61d190\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb"
Sep 11 18:29:18.116965 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:18.116907    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14a667c-e9e9-41ba-8bd7-105565a09431-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt\" (UID: \"e14a667c-e9e9-41ba-8bd7-105565a09431\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt"
Sep 11 18:29:18.124425 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:18.124393    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl7cj\" (UniqueName: \"kubernetes.io/projected/e14a667c-e9e9-41ba-8bd7-105565a09431-kube-api-access-tl7cj\") pod \"payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt\" (UID: \"e14a667c-e9e9-41ba-8bd7-105565a09431\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt"
Sep 11 18:29:18.124536 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:18.124424    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zv8v\" (UniqueName: \"kubernetes.io/projected/6546fdce-36a2-4a07-a06c-4786da61d190-kube-api-access-9zv8v\") pod \"payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb\" (UID: \"6546fdce-36a2-4a07-a06c-4786da61d190\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb"
Sep 11 18:29:18.218560 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:18.218518    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt"
Sep 11 18:29:18.248978 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:18.248949    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb"
Sep 11 18:29:18.494804 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:18.494778    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt"]
Sep 11 18:29:18.496880 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:29:18.496856    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14a667c_e9e9_41ba_8bd7_105565a09431.slice/crio-4265cd45c64a188bfdcef4c5c64819ebb3845265c52a4982daac8960be5f1e43 WatchSource:0}: Error finding container 4265cd45c64a188bfdcef4c5c64819ebb3845265c52a4982daac8960be5f1e43: Status 404 returned error can't find the container with id 4265cd45c64a188bfdcef4c5c64819ebb3845265c52a4982daac8960be5f1e43
Sep 11 18:29:18.529033 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:18.529006    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb"]
Sep 11 18:29:18.530461 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:29:18.530433    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6546fdce_36a2_4a07_a06c_4786da61d190.slice/crio-c00076b6c3b22cb479f162bb46b8ca3121c65ece67eaef620cd82c3e58a61a2e WatchSource:0}: Error finding container c00076b6c3b22cb479f162bb46b8ca3121c65ece67eaef620cd82c3e58a61a2e: Status 404 returned error can't find the container with id c00076b6c3b22cb479f162bb46b8ca3121c65ece67eaef620cd82c3e58a61a2e
Sep 11 18:29:18.861971 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:18.861866    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb" event={"ID":"6546fdce-36a2-4a07-a06c-4786da61d190","Type":"ContainerStarted","Data":"1def032bfe962225706e52bc8fa8a078b5c72da5bda5ff6357a26f64322e6365"}
Sep 11 18:29:18.861971 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:18.861929    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb" event={"ID":"6546fdce-36a2-4a07-a06c-4786da61d190","Type":"ContainerStarted","Data":"c00076b6c3b22cb479f162bb46b8ca3121c65ece67eaef620cd82c3e58a61a2e"}
Sep 11 18:29:18.862182 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:18.862058    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb"
Sep 11 18:29:18.863569 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:18.863541    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt" event={"ID":"e14a667c-e9e9-41ba-8bd7-105565a09431","Type":"ContainerStarted","Data":"2968876b30887f1771508613e59cb3f8bd85648de170f40a51146952e78be5c5"}
Sep 11 18:29:18.863704 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:18.863600    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt" event={"ID":"e14a667c-e9e9-41ba-8bd7-105565a09431","Type":"ContainerStarted","Data":"4265cd45c64a188bfdcef4c5c64819ebb3845265c52a4982daac8960be5f1e43"}
Sep 11 18:29:18.863704 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:18.863631    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt"
Sep 11 18:29:18.886829 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:18.886777    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb" podStartSLOduration=1.886757565 podStartE2EDuration="1.886757565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:29:18.884847288 +0000 UTC m=+1157.486208493" watchObservedRunningTime="2026-09-11 18:29:18.886757565 +0000 UTC m=+1157.488118733"
Sep 11 18:29:18.910787 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:18.910737    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt" podStartSLOduration=1.9107221110000001 podStartE2EDuration="1.910722111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:29:18.908971109 +0000 UTC m=+1157.510332274" watchObservedRunningTime="2026-09-11 18:29:18.910722111 +0000 UTC m=+1157.512083274"
Sep 11 18:29:26.019670 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:26.019570    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:29:26.019670 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:26.019644    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:29:26.032118 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:26.032087    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:29:26.914370 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:26.914337    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:29:29.869543 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:29.869513    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt"
Sep 11 18:29:29.869913 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:29.869639    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb"
Sep 11 18:29:38.183539 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:38.183499    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"]
Sep 11 18:29:47.731903 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:47.731865    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:29:51.590491 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:51.590457    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4"]
Sep 11 18:29:51.590901 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:51.590696    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4" podUID="092d7b84-cf77-4b84-b5cb-01b0b01a9a75" containerName="maas-api" containerID="cri-o://454ef1ee6eec3d11f43e020ba75041d6d1e205c8091f7ee6baa2c1e9a083f9cd" gracePeriod=30
Sep 11 18:29:51.644923 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:51.644888    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8"]
Sep 11 18:29:51.645188 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:51.645125    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8" podUID="56b76440-2d65-4538-9cf9-72c2fdaf3f1c" containerName="payload-processing" containerID="cri-o://2705c6c9f9087120fb74b57397e36cd687201d5df5edfaa984b04eade516d4aa" gracePeriod=30
Sep 11 18:29:51.895021 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:51.894997    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4"
Sep 11 18:29:51.921050 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:51.921017    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8"
Sep 11 18:29:51.926375 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:51.926341    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/092d7b84-cf77-4b84-b5cb-01b0b01a9a75-openshift-service-ca\") pod \"092d7b84-cf77-4b84-b5cb-01b0b01a9a75\" (UID: \"092d7b84-cf77-4b84-b5cb-01b0b01a9a75\") "
Sep 11 18:29:51.926528 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:51.926388    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfbnm\" (UniqueName: \"kubernetes.io/projected/092d7b84-cf77-4b84-b5cb-01b0b01a9a75-kube-api-access-sfbnm\") pod \"092d7b84-cf77-4b84-b5cb-01b0b01a9a75\" (UID: \"092d7b84-cf77-4b84-b5cb-01b0b01a9a75\") "
Sep 11 18:29:51.926528 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:51.926468    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/092d7b84-cf77-4b84-b5cb-01b0b01a9a75-metrics-certs\") pod \"092d7b84-cf77-4b84-b5cb-01b0b01a9a75\" (UID: \"092d7b84-cf77-4b84-b5cb-01b0b01a9a75\") "
Sep 11 18:29:51.926528 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:51.926493    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/092d7b84-cf77-4b84-b5cb-01b0b01a9a75-maas-api-tls\") pod \"092d7b84-cf77-4b84-b5cb-01b0b01a9a75\" (UID: \"092d7b84-cf77-4b84-b5cb-01b0b01a9a75\") "
Sep 11 18:29:51.926844 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:51.926816    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/092d7b84-cf77-4b84-b5cb-01b0b01a9a75-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "092d7b84-cf77-4b84-b5cb-01b0b01a9a75" (UID: "092d7b84-cf77-4b84-b5cb-01b0b01a9a75"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:29:51.927158 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:51.927135    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/092d7b84-cf77-4b84-b5cb-01b0b01a9a75-openshift-service-ca\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:29:51.928877 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:51.928846    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/092d7b84-cf77-4b84-b5cb-01b0b01a9a75-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "092d7b84-cf77-4b84-b5cb-01b0b01a9a75" (UID: "092d7b84-cf77-4b84-b5cb-01b0b01a9a75"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:29:51.928877 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:51.928860    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/092d7b84-cf77-4b84-b5cb-01b0b01a9a75-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "092d7b84-cf77-4b84-b5cb-01b0b01a9a75" (UID: "092d7b84-cf77-4b84-b5cb-01b0b01a9a75"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:29:51.929005 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:51.928962    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092d7b84-cf77-4b84-b5cb-01b0b01a9a75-kube-api-access-sfbnm" (OuterVolumeSpecName: "kube-api-access-sfbnm") pod "092d7b84-cf77-4b84-b5cb-01b0b01a9a75" (UID: "092d7b84-cf77-4b84-b5cb-01b0b01a9a75"). InnerVolumeSpecName "kube-api-access-sfbnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:29:52.024977 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.024943    2576 generic.go:356] "Generic (PLEG): container finished" podID="56b76440-2d65-4538-9cf9-72c2fdaf3f1c" containerID="2705c6c9f9087120fb74b57397e36cd687201d5df5edfaa984b04eade516d4aa" exitCode=0
Sep 11 18:29:52.025191 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.024983    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8"
Sep 11 18:29:52.025191 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.025025    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8" event={"ID":"56b76440-2d65-4538-9cf9-72c2fdaf3f1c","Type":"ContainerDied","Data":"2705c6c9f9087120fb74b57397e36cd687201d5df5edfaa984b04eade516d4aa"}
Sep 11 18:29:52.025191 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.025078    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8" event={"ID":"56b76440-2d65-4538-9cf9-72c2fdaf3f1c","Type":"ContainerDied","Data":"7d097fe687edc7f5452e978580814a00238c9cc47d7e2909b8f724b03dba92ca"}
Sep 11 18:29:52.025191 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.025108    2576 scope.go:117] "RemoveContainer" containerID="2705c6c9f9087120fb74b57397e36cd687201d5df5edfaa984b04eade516d4aa"
Sep 11 18:29:52.026469 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.026444    2576 generic.go:356] "Generic (PLEG): container finished" podID="092d7b84-cf77-4b84-b5cb-01b0b01a9a75" containerID="454ef1ee6eec3d11f43e020ba75041d6d1e205c8091f7ee6baa2c1e9a083f9cd" exitCode=0
Sep 11 18:29:52.026614 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.026488    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4"
Sep 11 18:29:52.026614 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.026494    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4" event={"ID":"092d7b84-cf77-4b84-b5cb-01b0b01a9a75","Type":"ContainerDied","Data":"454ef1ee6eec3d11f43e020ba75041d6d1e205c8091f7ee6baa2c1e9a083f9cd"}
Sep 11 18:29:52.026614 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.026527    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4" event={"ID":"092d7b84-cf77-4b84-b5cb-01b0b01a9a75","Type":"ContainerDied","Data":"c9b6e3216a76e4bacee383fb3b4738008c81eb4d39c4ba3ad1a998b64c94640e"}
Sep 11 18:29:52.027894 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.027870    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24fmp\" (UniqueName: \"kubernetes.io/projected/56b76440-2d65-4538-9cf9-72c2fdaf3f1c-kube-api-access-24fmp\") pod \"56b76440-2d65-4538-9cf9-72c2fdaf3f1c\" (UID: \"56b76440-2d65-4538-9cf9-72c2fdaf3f1c\") "
Sep 11 18:29:52.027995 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.027935    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b76440-2d65-4538-9cf9-72c2fdaf3f1c-plugins-config-volume\") pod \"56b76440-2d65-4538-9cf9-72c2fdaf3f1c\" (UID: \"56b76440-2d65-4538-9cf9-72c2fdaf3f1c\") "
Sep 11 18:29:52.028193 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.028175    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfbnm\" (UniqueName: \"kubernetes.io/projected/092d7b84-cf77-4b84-b5cb-01b0b01a9a75-kube-api-access-sfbnm\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:29:52.028261 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.028198    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/092d7b84-cf77-4b84-b5cb-01b0b01a9a75-metrics-certs\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:29:52.028261 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.028212    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/092d7b84-cf77-4b84-b5cb-01b0b01a9a75-maas-api-tls\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:29:52.028342 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.028291    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56b76440-2d65-4538-9cf9-72c2fdaf3f1c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56b76440-2d65-4538-9cf9-72c2fdaf3f1c" (UID: "56b76440-2d65-4538-9cf9-72c2fdaf3f1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:29:52.030263 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.030238    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b76440-2d65-4538-9cf9-72c2fdaf3f1c-kube-api-access-24fmp" (OuterVolumeSpecName: "kube-api-access-24fmp") pod "56b76440-2d65-4538-9cf9-72c2fdaf3f1c" (UID: "56b76440-2d65-4538-9cf9-72c2fdaf3f1c"). InnerVolumeSpecName "kube-api-access-24fmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:29:52.034318 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.034302    2576 scope.go:117] "RemoveContainer" containerID="2705c6c9f9087120fb74b57397e36cd687201d5df5edfaa984b04eade516d4aa"
Sep 11 18:29:52.034671 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:29:52.034649    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2705c6c9f9087120fb74b57397e36cd687201d5df5edfaa984b04eade516d4aa\": container with ID starting with 2705c6c9f9087120fb74b57397e36cd687201d5df5edfaa984b04eade516d4aa not found: ID does not exist" containerID="2705c6c9f9087120fb74b57397e36cd687201d5df5edfaa984b04eade516d4aa"
Sep 11 18:29:52.034750 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.034682    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2705c6c9f9087120fb74b57397e36cd687201d5df5edfaa984b04eade516d4aa"} err="failed to get container status \"2705c6c9f9087120fb74b57397e36cd687201d5df5edfaa984b04eade516d4aa\": rpc error: code = NotFound desc = could not find container \"2705c6c9f9087120fb74b57397e36cd687201d5df5edfaa984b04eade516d4aa\": container with ID starting with 2705c6c9f9087120fb74b57397e36cd687201d5df5edfaa984b04eade516d4aa not found: ID does not exist"
Sep 11 18:29:52.034750 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.034699    2576 scope.go:117] "RemoveContainer" containerID="454ef1ee6eec3d11f43e020ba75041d6d1e205c8091f7ee6baa2c1e9a083f9cd"
Sep 11 18:29:52.044425 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.044407    2576 scope.go:117] "RemoveContainer" containerID="454ef1ee6eec3d11f43e020ba75041d6d1e205c8091f7ee6baa2c1e9a083f9cd"
Sep 11 18:29:52.044723 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:29:52.044703    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"454ef1ee6eec3d11f43e020ba75041d6d1e205c8091f7ee6baa2c1e9a083f9cd\": container with ID starting with 454ef1ee6eec3d11f43e020ba75041d6d1e205c8091f7ee6baa2c1e9a083f9cd not found: ID does not exist" containerID="454ef1ee6eec3d11f43e020ba75041d6d1e205c8091f7ee6baa2c1e9a083f9cd"
Sep 11 18:29:52.044801 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.044727    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454ef1ee6eec3d11f43e020ba75041d6d1e205c8091f7ee6baa2c1e9a083f9cd"} err="failed to get container status \"454ef1ee6eec3d11f43e020ba75041d6d1e205c8091f7ee6baa2c1e9a083f9cd\": rpc error: code = NotFound desc = could not find container \"454ef1ee6eec3d11f43e020ba75041d6d1e205c8091f7ee6baa2c1e9a083f9cd\": container with ID starting with 454ef1ee6eec3d11f43e020ba75041d6d1e205c8091f7ee6baa2c1e9a083f9cd not found: ID does not exist"
Sep 11 18:29:52.055727 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.055693    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4"]
Sep 11 18:29:52.071028 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.070999    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f63893-5545d46d88-442t4"]
Sep 11 18:29:52.129672 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.129565    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24fmp\" (UniqueName: \"kubernetes.io/projected/56b76440-2d65-4538-9cf9-72c2fdaf3f1c-kube-api-access-24fmp\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:29:52.129672 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.129616    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b76440-2d65-4538-9cf9-72c2fdaf3f1c-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:29:52.345018 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.344982    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8"]
Sep 11 18:29:52.351819 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.351787    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f63893-6c55b6746c-2g9q8"]
Sep 11 18:29:52.758408 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.758371    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h"]
Sep 11 18:29:52.758811 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.758772    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="092d7b84-cf77-4b84-b5cb-01b0b01a9a75" containerName="maas-api"
Sep 11 18:29:52.758811 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.758785    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="092d7b84-cf77-4b84-b5cb-01b0b01a9a75" containerName="maas-api"
Sep 11 18:29:52.758811 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.758798    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56b76440-2d65-4538-9cf9-72c2fdaf3f1c" containerName="payload-processing"
Sep 11 18:29:52.758811 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.758804    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="56b76440-2d65-4538-9cf9-72c2fdaf3f1c" containerName="payload-processing"
Sep 11 18:29:52.758985 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.758861    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="56b76440-2d65-4538-9cf9-72c2fdaf3f1c" containerName="payload-processing"
Sep 11 18:29:52.758985 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.758872    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="092d7b84-cf77-4b84-b5cb-01b0b01a9a75" containerName="maas-api"
Sep 11 18:29:52.763393 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.763374    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h"
Sep 11 18:29:52.765493 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.765474    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-6c3180-dockercfg-jk4tr\""
Sep 11 18:29:52.765561 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.765533    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-6c3180\""
Sep 11 18:29:52.773615 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.773571    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h"]
Sep 11 18:29:52.836037 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.836003    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b3a597-b5c6-4da5-914d-4c4ed2781b1e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h\" (UID: \"a1b3a597-b5c6-4da5-914d-4c4ed2781b1e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h"
Sep 11 18:29:52.836037 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.836040    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqrdn\" (UniqueName: \"kubernetes.io/projected/a1b3a597-b5c6-4da5-914d-4c4ed2781b1e-kube-api-access-pqrdn\") pod \"payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h\" (UID: \"a1b3a597-b5c6-4da5-914d-4c4ed2781b1e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h"
Sep 11 18:29:52.937635 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.937509    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b3a597-b5c6-4da5-914d-4c4ed2781b1e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h\" (UID: \"a1b3a597-b5c6-4da5-914d-4c4ed2781b1e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h"
Sep 11 18:29:52.937635 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.937563    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqrdn\" (UniqueName: \"kubernetes.io/projected/a1b3a597-b5c6-4da5-914d-4c4ed2781b1e-kube-api-access-pqrdn\") pod \"payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h\" (UID: \"a1b3a597-b5c6-4da5-914d-4c4ed2781b1e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h"
Sep 11 18:29:52.938282 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.938260    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b3a597-b5c6-4da5-914d-4c4ed2781b1e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h\" (UID: \"a1b3a597-b5c6-4da5-914d-4c4ed2781b1e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h"
Sep 11 18:29:52.945831 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:52.945798    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqrdn\" (UniqueName: \"kubernetes.io/projected/a1b3a597-b5c6-4da5-914d-4c4ed2781b1e-kube-api-access-pqrdn\") pod \"payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h\" (UID: \"a1b3a597-b5c6-4da5-914d-4c4ed2781b1e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h"
Sep 11 18:29:53.074943 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:53.074910    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h"
Sep 11 18:29:53.203525 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:53.203485    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g" podUID="38a9a26d-5f5a-4cf6-928a-cf5be2e04e40" containerName="main" containerID="cri-o://85cb051e23254814dd0736005a1b41074907064710ea2364da8581eadde83266" gracePeriod=45
Sep 11 18:29:53.350566 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:53.350538    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h"]
Sep 11 18:29:53.353239 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:29:53.353202    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b3a597_b5c6_4da5_914d_4c4ed2781b1e.slice/crio-49ac038e7e78f1ce3dfcd717b4e18a14361b89b3d94ded7f9002961725be76a6 WatchSource:0}: Error finding container 49ac038e7e78f1ce3dfcd717b4e18a14361b89b3d94ded7f9002961725be76a6: Status 404 returned error can't find the container with id 49ac038e7e78f1ce3dfcd717b4e18a14361b89b3d94ded7f9002961725be76a6
Sep 11 18:29:53.452716 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:53.452686    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:29:53.544659 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:53.544560    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-tls-certs\") pod \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\" (UID: \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\") "
Sep 11 18:29:53.544659 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:53.544626    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-home\") pod \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\" (UID: \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\") "
Sep 11 18:29:53.544910 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:53.544673    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-dshm\") pod \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\" (UID: \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\") "
Sep 11 18:29:53.544910 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:53.544765    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-model-cache\") pod \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\" (UID: \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\") "
Sep 11 18:29:53.544910 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:53.544813    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-tmp-dir\") pod \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\" (UID: \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\") "
Sep 11 18:29:53.544910 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:53.544867    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqp6h\" (UniqueName: \"kubernetes.io/projected/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-kube-api-access-nqp6h\") pod \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\" (UID: \"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40\") "
Sep 11 18:29:53.545117 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:53.544924    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-home" (OuterVolumeSpecName: "home") pod "38a9a26d-5f5a-4cf6-928a-cf5be2e04e40" (UID: "38a9a26d-5f5a-4cf6-928a-cf5be2e04e40"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:29:53.545117 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:53.544970    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-model-cache" (OuterVolumeSpecName: "model-cache") pod "38a9a26d-5f5a-4cf6-928a-cf5be2e04e40" (UID: "38a9a26d-5f5a-4cf6-928a-cf5be2e04e40"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:29:53.545117 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:53.545097    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38a9a26d-5f5a-4cf6-928a-cf5be2e04e40" (UID: "38a9a26d-5f5a-4cf6-928a-cf5be2e04e40"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:29:53.545258 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:53.545236    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-home\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:29:53.545316 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:53.545255    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-model-cache\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:29:53.545316 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:53.545294    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-tmp-dir\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:29:53.547024 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:53.547000    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-kube-api-access-nqp6h" (OuterVolumeSpecName: "kube-api-access-nqp6h") pod "38a9a26d-5f5a-4cf6-928a-cf5be2e04e40" (UID: "38a9a26d-5f5a-4cf6-928a-cf5be2e04e40"). InnerVolumeSpecName "kube-api-access-nqp6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:29:53.547238 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:53.547217    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38a9a26d-5f5a-4cf6-928a-cf5be2e04e40" (UID: "38a9a26d-5f5a-4cf6-928a-cf5be2e04e40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:29:53.547301 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:53.547277    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-dshm" (OuterVolumeSpecName: "dshm") pod "38a9a26d-5f5a-4cf6-928a-cf5be2e04e40" (UID: "38a9a26d-5f5a-4cf6-928a-cf5be2e04e40"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:29:53.646700 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:53.646657    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-tls-certs\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:29:53.646700 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:53.646693    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-dshm\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:29:53.646700 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:53.646705    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqp6h\" (UniqueName: \"kubernetes.io/projected/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40-kube-api-access-nqp6h\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:29:53.910852 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:53.910768    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="092d7b84-cf77-4b84-b5cb-01b0b01a9a75" path="/var/lib/kubelet/pods/092d7b84-cf77-4b84-b5cb-01b0b01a9a75/volumes"
Sep 11 18:29:53.911190 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:53.911136    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56b76440-2d65-4538-9cf9-72c2fdaf3f1c" path="/var/lib/kubelet/pods/56b76440-2d65-4538-9cf9-72c2fdaf3f1c/volumes"
Sep 11 18:29:54.039111 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:54.039077    2576 generic.go:356] "Generic (PLEG): container finished" podID="38a9a26d-5f5a-4cf6-928a-cf5be2e04e40" containerID="85cb051e23254814dd0736005a1b41074907064710ea2364da8581eadde83266" exitCode=0
Sep 11 18:29:54.039308 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:54.039130    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"
Sep 11 18:29:54.039308 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:54.039178    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g" event={"ID":"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40","Type":"ContainerDied","Data":"85cb051e23254814dd0736005a1b41074907064710ea2364da8581eadde83266"}
Sep 11 18:29:54.039308 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:54.039231    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g" event={"ID":"38a9a26d-5f5a-4cf6-928a-cf5be2e04e40","Type":"ContainerDied","Data":"6f8f826473b26b9e2533aa21c116c7787dac60597898f53f2fb569a2e331f385"}
Sep 11 18:29:54.039308 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:54.039254    2576 scope.go:117] "RemoveContainer" containerID="85cb051e23254814dd0736005a1b41074907064710ea2364da8581eadde83266"
Sep 11 18:29:54.041647 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:54.041607    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h" event={"ID":"a1b3a597-b5c6-4da5-914d-4c4ed2781b1e","Type":"ContainerStarted","Data":"a2998737d354526b7ba962252ae74f6fb92f5b2a8f8c1807e9e3e35d8ddf91ac"}
Sep 11 18:29:54.041791 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:54.041660    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h" event={"ID":"a1b3a597-b5c6-4da5-914d-4c4ed2781b1e","Type":"ContainerStarted","Data":"49ac038e7e78f1ce3dfcd717b4e18a14361b89b3d94ded7f9002961725be76a6"}
Sep 11 18:29:54.041874 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:54.041851    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h"
Sep 11 18:29:54.048002 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:54.047983    2576 scope.go:117] "RemoveContainer" containerID="85cb051e23254814dd0736005a1b41074907064710ea2364da8581eadde83266"
Sep 11 18:29:54.048295 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:29:54.048278    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85cb051e23254814dd0736005a1b41074907064710ea2364da8581eadde83266\": container with ID starting with 85cb051e23254814dd0736005a1b41074907064710ea2364da8581eadde83266 not found: ID does not exist" containerID="85cb051e23254814dd0736005a1b41074907064710ea2364da8581eadde83266"
Sep 11 18:29:54.048348 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:54.048304    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85cb051e23254814dd0736005a1b41074907064710ea2364da8581eadde83266"} err="failed to get container status \"85cb051e23254814dd0736005a1b41074907064710ea2364da8581eadde83266\": rpc error: code = NotFound desc = could not find container \"85cb051e23254814dd0736005a1b41074907064710ea2364da8581eadde83266\": container with ID starting with 85cb051e23254814dd0736005a1b41074907064710ea2364da8581eadde83266 not found: ID does not exist"
Sep 11 18:29:54.060622 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:54.060538    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h" podStartSLOduration=2.060522584 podStartE2EDuration="2.060522584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:29:54.058303028 +0000 UTC m=+1192.659664195" watchObservedRunningTime="2026-09-11 18:29:54.060522584 +0000 UTC m=+1192.661883747"
Sep 11 18:29:54.074336 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:54.074306    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"]
Sep 11 18:29:54.078430 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:54.078403    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f63893/test-model-f63893-kserve-7878589b9f-w5v7g"]
Sep 11 18:29:55.911733 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:29:55.911698    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38a9a26d-5f5a-4cf6-928a-cf5be2e04e40" path="/var/lib/kubelet/pods/38a9a26d-5f5a-4cf6-928a-cf5be2e04e40/volumes"
Sep 11 18:30:00.143082 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:00.143047    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819190-zfz2l"]
Sep 11 18:30:00.143485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:00.143400    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38a9a26d-5f5a-4cf6-928a-cf5be2e04e40" containerName="main"
Sep 11 18:30:00.143485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:00.143411    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="38a9a26d-5f5a-4cf6-928a-cf5be2e04e40" containerName="main"
Sep 11 18:30:00.143485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:00.143477    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="38a9a26d-5f5a-4cf6-928a-cf5be2e04e40" containerName="main"
Sep 11 18:30:00.148029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:00.148012    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-zfz2l"
Sep 11 18:30:00.150226 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:00.150207    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t9kdx\""
Sep 11 18:30:00.156515 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:00.156489    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819190-zfz2l"]
Sep 11 18:30:00.159450 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:00.159428    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-zfz2l"
Sep 11 18:30:00.167687 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:00.167663    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c3180-29819190-gjpzf"]
Sep 11 18:30:00.171175 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:00.171151    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c3180-29819190-gjpzf"
Sep 11 18:30:00.179826 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:00.179795    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c3180-29819190-gjpzf"]
Sep 11 18:30:00.194885 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:00.194859    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c3180-29819190-gjpzf"
Sep 11 18:30:00.438354 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:00.438266    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819190-zfz2l"]
Sep 11 18:30:00.442267 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:30:00.442234    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0d9e44_c87f_49b4_9d30_15be0ba26458.slice/crio-ed780b5249bb189886e92cba585c0b93bf5c1186e9e87dd7caae7517b5d06c18 WatchSource:0}: Error finding container ed780b5249bb189886e92cba585c0b93bf5c1186e9e87dd7caae7517b5d06c18: Status 404 returned error can't find the container with id ed780b5249bb189886e92cba585c0b93bf5c1186e9e87dd7caae7517b5d06c18
Sep 11 18:30:00.444164 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:00.444143    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:30:00.478094 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:00.478062    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c3180-29819190-gjpzf"]
Sep 11 18:30:00.480577 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:30:00.480549    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24483d4_3bd4_4d9d_b40f_305d28b9586a.slice/crio-a63a91d157a8894c6977630187139d47d9f19b8bcee8e9e34d1269af3bb7fa3c WatchSource:0}: Error finding container a63a91d157a8894c6977630187139d47d9f19b8bcee8e9e34d1269af3bb7fa3c: Status 404 returned error can't find the container with id a63a91d157a8894c6977630187139d47d9f19b8bcee8e9e34d1269af3bb7fa3c
Sep 11 18:30:01.077300 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:01.077262    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c3180-29819190-gjpzf" event={"ID":"e24483d4-3bd4-4d9d-b40f-305d28b9586a","Type":"ContainerStarted","Data":"a63a91d157a8894c6977630187139d47d9f19b8bcee8e9e34d1269af3bb7fa3c"}
Sep 11 18:30:01.078482 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:01.078456    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-zfz2l" event={"ID":"eb0d9e44-c87f-49b4-9d30-15be0ba26458","Type":"ContainerStarted","Data":"ed780b5249bb189886e92cba585c0b93bf5c1186e9e87dd7caae7517b5d06c18"}
Sep 11 18:30:03.091377 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:03.091335    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c3180-29819190-gjpzf" event={"ID":"e24483d4-3bd4-4d9d-b40f-305d28b9586a","Type":"ContainerStarted","Data":"cd9625728d4acb2b1335aa9d0d45d26ad585133eba7f8767684840280418dcf2"}
Sep 11 18:30:03.092828 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:03.092802    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-zfz2l" event={"ID":"eb0d9e44-c87f-49b4-9d30-15be0ba26458","Type":"ContainerStarted","Data":"acb002060085407638d6d27792118dc67dc2828be2b317c2cccb578fd012a1ca"}
Sep 11 18:30:03.110114 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:03.110064    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c3180-29819190-gjpzf" podStartSLOduration=0.829140465 podStartE2EDuration="3.110044926s" totalImagesPullingTime="2.280904461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:03.109072923 +0000 UTC m=+1201.710434087" watchObservedRunningTime="2026-09-11 18:30:03.110044926 +0000 UTC m=+1201.711406079"
Sep 11 18:30:03.127190 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:03.127138    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-zfz2l" podStartSLOduration=0.812502602 podStartE2EDuration="3.127120799s" totalImagesPullingTime="2.314618197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:03.126880503 +0000 UTC m=+1201.728241665" watchObservedRunningTime="2026-09-11 18:30:03.127120799 +0000 UTC m=+1201.728481962"
Sep 11 18:30:04.098888 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:04.098796    2576 generic.go:356] "Generic (PLEG): container finished" podID="e24483d4-3bd4-4d9d-b40f-305d28b9586a" containerID="cd9625728d4acb2b1335aa9d0d45d26ad585133eba7f8767684840280418dcf2" exitCode=0
Sep 11 18:30:04.099308 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:04.098882    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c3180-29819190-gjpzf" event={"ID":"e24483d4-3bd4-4d9d-b40f-305d28b9586a","Type":"ContainerDied","Data":"cd9625728d4acb2b1335aa9d0d45d26ad585133eba7f8767684840280418dcf2"}
Sep 11 18:30:04.100420 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:04.100394    2576 generic.go:356] "Generic (PLEG): container finished" podID="eb0d9e44-c87f-49b4-9d30-15be0ba26458" containerID="acb002060085407638d6d27792118dc67dc2828be2b317c2cccb578fd012a1ca" exitCode=0
Sep 11 18:30:04.100522 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:04.100466    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-zfz2l" event={"ID":"eb0d9e44-c87f-49b4-9d30-15be0ba26458","Type":"ContainerDied","Data":"acb002060085407638d6d27792118dc67dc2828be2b317c2cccb578fd012a1ca"}
Sep 11 18:30:05.049775 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:05.049736    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h"
Sep 11 18:30:05.257036 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:05.257003    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-zfz2l"
Sep 11 18:30:05.288072 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:05.288044    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c3180-29819190-gjpzf"
Sep 11 18:30:06.112061 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:06.112028    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c3180-29819190-gjpzf"
Sep 11 18:30:06.112269 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:06.112056    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c3180-29819190-gjpzf" event={"ID":"e24483d4-3bd4-4d9d-b40f-305d28b9586a","Type":"ContainerDied","Data":"a63a91d157a8894c6977630187139d47d9f19b8bcee8e9e34d1269af3bb7fa3c"}
Sep 11 18:30:06.112269 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:06.112122    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a63a91d157a8894c6977630187139d47d9f19b8bcee8e9e34d1269af3bb7fa3c"
Sep 11 18:30:06.113594 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:06.113556    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-zfz2l"
Sep 11 18:30:06.113721 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:06.113602    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-zfz2l" event={"ID":"eb0d9e44-c87f-49b4-9d30-15be0ba26458","Type":"ContainerDied","Data":"ed780b5249bb189886e92cba585c0b93bf5c1186e9e87dd7caae7517b5d06c18"}
Sep 11 18:30:06.113721 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:06.113642    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed780b5249bb189886e92cba585c0b93bf5c1186e9e87dd7caae7517b5d06c18"
Sep 11 18:30:08.181420 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:08.181379    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7839-3b6d08"]
Sep 11 18:30:08.181956 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:08.181935    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb0d9e44-c87f-49b4-9d30-15be0ba26458" containerName="cleanup"
Sep 11 18:30:08.182031 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:08.181960    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb0d9e44-c87f-49b4-9d30-15be0ba26458" containerName="cleanup"
Sep 11 18:30:08.182031 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:08.181988    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e24483d4-3bd4-4d9d-b40f-305d28b9586a" containerName="cleanup"
Sep 11 18:30:08.182031 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:08.181997    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e24483d4-3bd4-4d9d-b40f-305d28b9586a" containerName="cleanup"
Sep 11 18:30:08.182173 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:08.182117    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb0d9e44-c87f-49b4-9d30-15be0ba26458" containerName="cleanup"
Sep 11 18:30:08.182173 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:08.182132    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e24483d4-3bd4-4d9d-b40f-305d28b9586a" containerName="cleanup"
Sep 11 18:30:08.185204 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:08.185180    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7839-3b6d08"
Sep 11 18:30:08.187325 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:08.187305    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fz6cx\""
Sep 11 18:30:08.192034 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:08.192006    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7839-3b6d08"]
Sep 11 18:30:08.275321 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:08.275271    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zx94\" (UniqueName: \"kubernetes.io/projected/aa7af295-5505-4a32-927e-94d5928802cf-kube-api-access-4zx94\") pod \"test-curl-7839-3b6d08\" (UID: \"aa7af295-5505-4a32-927e-94d5928802cf\") " pod="opendatahub/test-curl-7839-3b6d08"
Sep 11 18:30:08.376523 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:08.376479    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zx94\" (UniqueName: \"kubernetes.io/projected/aa7af295-5505-4a32-927e-94d5928802cf-kube-api-access-4zx94\") pod \"test-curl-7839-3b6d08\" (UID: \"aa7af295-5505-4a32-927e-94d5928802cf\") " pod="opendatahub/test-curl-7839-3b6d08"
Sep 11 18:30:08.384339 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:08.384300    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zx94\" (UniqueName: \"kubernetes.io/projected/aa7af295-5505-4a32-927e-94d5928802cf-kube-api-access-4zx94\") pod \"test-curl-7839-3b6d08\" (UID: \"aa7af295-5505-4a32-927e-94d5928802cf\") " pod="opendatahub/test-curl-7839-3b6d08"
Sep 11 18:30:08.495224 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:08.495182    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7839-3b6d08"
Sep 11 18:30:08.765529 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:08.765499    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7839-3b6d08"]
Sep 11 18:30:08.767860 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:30:08.767832    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa7af295_5505_4a32_927e_94d5928802cf.slice/crio-ca189e5a4a4003bb9eedc0af3c7279c6aeacec9909f8ea498b2c5c6c9c4ca01e WatchSource:0}: Error finding container ca189e5a4a4003bb9eedc0af3c7279c6aeacec9909f8ea498b2c5c6c9c4ca01e: Status 404 returned error can't find the container with id ca189e5a4a4003bb9eedc0af3c7279c6aeacec9909f8ea498b2c5c6c9c4ca01e
Sep 11 18:30:09.130608 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:09.130495    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7839-3b6d08" event={"ID":"aa7af295-5505-4a32-927e-94d5928802cf","Type":"ContainerStarted","Data":"55c0c541f77abacf809a23fc17276a82dfb92f45e9102e45e42fdd7bcc531bc0"}
Sep 11 18:30:09.130608 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:09.130544    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7839-3b6d08" event={"ID":"aa7af295-5505-4a32-927e-94d5928802cf","Type":"ContainerStarted","Data":"ca189e5a4a4003bb9eedc0af3c7279c6aeacec9909f8ea498b2c5c6c9c4ca01e"}
Sep 11 18:30:09.145353 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:09.145301    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7839-3b6d08" podStartSLOduration=1.145283278 podStartE2EDuration="1.145283278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:09.144154107 +0000 UTC m=+1207.745515252" watchObservedRunningTime="2026-09-11 18:30:09.145283278 +0000 UTC m=+1207.746644441"
Sep 11 18:30:09.713699 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:09.713660    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7839-3b6d08"]
Sep 11 18:30:09.717690 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:09.717663    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7839-3b6d08"]
Sep 11 18:30:09.832197 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:09.832102    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7839-87d585"]
Sep 11 18:30:09.832566 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:09.832548    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa7af295-5505-4a32-927e-94d5928802cf" containerName="test-curl-7839-3b6d08"
Sep 11 18:30:09.832632 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:09.832569    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa7af295-5505-4a32-927e-94d5928802cf" containerName="test-curl-7839-3b6d08"
Sep 11 18:30:09.832670 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:09.832642    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa7af295-5505-4a32-927e-94d5928802cf" containerName="test-curl-7839-3b6d08"
Sep 11 18:30:09.835769 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:09.835748    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7839-87d585"
Sep 11 18:30:09.840176 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:09.840149    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7839-87d585"]
Sep 11 18:30:09.993077 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:09.993036    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd87t\" (UniqueName: \"kubernetes.io/projected/584b719a-d733-40d4-a6f4-5b4f86080f56-kube-api-access-zd87t\") pod \"test-curl-7839-87d585\" (UID: \"584b719a-d733-40d4-a6f4-5b4f86080f56\") " pod="opendatahub/test-curl-7839-87d585"
Sep 11 18:30:10.094665 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:10.094542    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd87t\" (UniqueName: \"kubernetes.io/projected/584b719a-d733-40d4-a6f4-5b4f86080f56-kube-api-access-zd87t\") pod \"test-curl-7839-87d585\" (UID: \"584b719a-d733-40d4-a6f4-5b4f86080f56\") " pod="opendatahub/test-curl-7839-87d585"
Sep 11 18:30:10.103427 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:10.103396    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd87t\" (UniqueName: \"kubernetes.io/projected/584b719a-d733-40d4-a6f4-5b4f86080f56-kube-api-access-zd87t\") pod \"test-curl-7839-87d585\" (UID: \"584b719a-d733-40d4-a6f4-5b4f86080f56\") " pod="opendatahub/test-curl-7839-87d585"
Sep 11 18:30:10.135491 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:30:10.135462    2576 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7839-3b6d08" container="test-curl-7839-3b6d08"
Sep 11 18:30:10.146840 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:10.146808    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7839-87d585"
Sep 11 18:30:10.418577 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:10.418544    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7839-87d585"]
Sep 11 18:30:10.420813 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:30:10.420783    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod584b719a_d733_40d4_a6f4_5b4f86080f56.slice/crio-a64580438610275d666446ad73263294f24eb7eaff1c7b79e6da090df525c30a WatchSource:0}: Error finding container a64580438610275d666446ad73263294f24eb7eaff1c7b79e6da090df525c30a: Status 404 returned error can't find the container with id a64580438610275d666446ad73263294f24eb7eaff1c7b79e6da090df525c30a
Sep 11 18:30:11.142068 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:11.142031    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7839-87d585" event={"ID":"584b719a-d733-40d4-a6f4-5b4f86080f56","Type":"ContainerStarted","Data":"00cfb22445865e63c910c2b7fce1d7ba194d1a9e43098f9bc3488d363e87e44b"}
Sep 11 18:30:11.142542 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:11.142081    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7839-87d585" event={"ID":"584b719a-d733-40d4-a6f4-5b4f86080f56","Type":"ContainerStarted","Data":"a64580438610275d666446ad73263294f24eb7eaff1c7b79e6da090df525c30a"}
Sep 11 18:30:11.142542 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:11.142180    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7839-3b6d08" podUID="aa7af295-5505-4a32-927e-94d5928802cf" containerName="test-curl-7839-3b6d08" containerID="cri-o://55c0c541f77abacf809a23fc17276a82dfb92f45e9102e45e42fdd7bcc531bc0" gracePeriod=2
Sep 11 18:30:11.158401 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:11.158346    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7839-87d585" podStartSLOduration=2.158328858 podStartE2EDuration="2.158328858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:11.1565669 +0000 UTC m=+1209.757928064" watchObservedRunningTime="2026-09-11 18:30:11.158328858 +0000 UTC m=+1209.759690021"
Sep 11 18:30:11.575032 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:11.574996    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"]
Sep 11 18:30:11.575277 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:11.575250    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6" podUID="8bf1b9de-9a29-4fbf-9ace-ea260daabbf8" containerName="maas-api" containerID="cri-o://5f57224806651c5e095de3d1ac54341ad707adc98d581a9de30cc289b0d847d4" gracePeriod=30
Sep 11 18:30:11.652577 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:11.652541    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2"]
Sep 11 18:30:11.652827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:11.652798    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2" podUID="3ac93d8a-158d-42b3-8b85-3217f8fab901" containerName="payload-pre-processing" containerID="cri-o://27e1c4d5299122e1bb748fd5a73f1f896a349e9c6a345c59dfe7f4cd5a24824d" gracePeriod=30
Sep 11 18:30:11.733192 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:11.733148    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7839-87d585"]
Sep 11 18:30:11.740853 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:11.740819    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7839-87d585"]
Sep 11 18:30:11.853307 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:11.853275    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7839-ab9061"]
Sep 11 18:30:11.853675 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:11.853658    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="584b719a-d733-40d4-a6f4-5b4f86080f56" containerName="test-curl-7839-87d585"
Sep 11 18:30:11.853738 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:11.853676    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="584b719a-d733-40d4-a6f4-5b4f86080f56" containerName="test-curl-7839-87d585"
Sep 11 18:30:11.853774 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:11.853743    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="584b719a-d733-40d4-a6f4-5b4f86080f56" containerName="test-curl-7839-87d585"
Sep 11 18:30:11.856780 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:11.856755    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7839-ab9061"
Sep 11 18:30:11.862856 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:11.862827    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7839-ab9061"]
Sep 11 18:30:11.907196 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:11.907161    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"
Sep 11 18:30:11.932078 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:11.932050    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2"
Sep 11 18:30:12.011384 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.011288    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-openshift-service-ca\") pod \"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8\" (UID: \"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8\") "
Sep 11 18:30:12.011384 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.011335    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbgdg\" (UniqueName: \"kubernetes.io/projected/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-kube-api-access-bbgdg\") pod \"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8\" (UID: \"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8\") "
Sep 11 18:30:12.011384 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.011374    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac93d8a-158d-42b3-8b85-3217f8fab901-plugins-config-volume\") pod \"3ac93d8a-158d-42b3-8b85-3217f8fab901\" (UID: \"3ac93d8a-158d-42b3-8b85-3217f8fab901\") "
Sep 11 18:30:12.011694 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.011424    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-metrics-certs\") pod \"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8\" (UID: \"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8\") "
Sep 11 18:30:12.011694 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.011480    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-maas-api-tls\") pod \"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8\" (UID: \"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8\") "
Sep 11 18:30:12.011694 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.011523    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qfh2\" (UniqueName: \"kubernetes.io/projected/3ac93d8a-158d-42b3-8b85-3217f8fab901-kube-api-access-7qfh2\") pod \"3ac93d8a-158d-42b3-8b85-3217f8fab901\" (UID: \"3ac93d8a-158d-42b3-8b85-3217f8fab901\") "
Sep 11 18:30:12.011694 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.011688    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdr8q\" (UniqueName: \"kubernetes.io/projected/0a0627ac-d50a-4442-a0c5-83157f3470a6-kube-api-access-bdr8q\") pod \"test-curl-7839-ab9061\" (UID: \"0a0627ac-d50a-4442-a0c5-83157f3470a6\") " pod="opendatahub/test-curl-7839-ab9061"
Sep 11 18:30:12.011916 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.011777    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8bf1b9de-9a29-4fbf-9ace-ea260daabbf8" (UID: "8bf1b9de-9a29-4fbf-9ace-ea260daabbf8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:30:12.011916 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.011808    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac93d8a-158d-42b3-8b85-3217f8fab901-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3ac93d8a-158d-42b3-8b85-3217f8fab901" (UID: "3ac93d8a-158d-42b3-8b85-3217f8fab901"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:30:12.012059 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.011928    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-openshift-service-ca\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:30:12.012059 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.011945    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac93d8a-158d-42b3-8b85-3217f8fab901-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:30:12.013578 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.013551    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-kube-api-access-bbgdg" (OuterVolumeSpecName: "kube-api-access-bbgdg") pod "8bf1b9de-9a29-4fbf-9ace-ea260daabbf8" (UID: "8bf1b9de-9a29-4fbf-9ace-ea260daabbf8"). InnerVolumeSpecName "kube-api-access-bbgdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:30:12.013674 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.013641    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8bf1b9de-9a29-4fbf-9ace-ea260daabbf8" (UID: "8bf1b9de-9a29-4fbf-9ace-ea260daabbf8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:30:12.014101 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.014079    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8bf1b9de-9a29-4fbf-9ace-ea260daabbf8" (UID: "8bf1b9de-9a29-4fbf-9ace-ea260daabbf8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:30:12.014140 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.014092    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac93d8a-158d-42b3-8b85-3217f8fab901-kube-api-access-7qfh2" (OuterVolumeSpecName: "kube-api-access-7qfh2") pod "3ac93d8a-158d-42b3-8b85-3217f8fab901" (UID: "3ac93d8a-158d-42b3-8b85-3217f8fab901"). InnerVolumeSpecName "kube-api-access-7qfh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:30:12.112785 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.112748    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdr8q\" (UniqueName: \"kubernetes.io/projected/0a0627ac-d50a-4442-a0c5-83157f3470a6-kube-api-access-bdr8q\") pod \"test-curl-7839-ab9061\" (UID: \"0a0627ac-d50a-4442-a0c5-83157f3470a6\") " pod="opendatahub/test-curl-7839-ab9061"
Sep 11 18:30:12.112928 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.112825    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-maas-api-tls\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:30:12.112928 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.112838    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qfh2\" (UniqueName: \"kubernetes.io/projected/3ac93d8a-158d-42b3-8b85-3217f8fab901-kube-api-access-7qfh2\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:30:12.112928 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.112848    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbgdg\" (UniqueName: \"kubernetes.io/projected/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-kube-api-access-bbgdg\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:30:12.112928 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.112861    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8-metrics-certs\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:30:12.121130 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.121103    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdr8q\" (UniqueName: \"kubernetes.io/projected/0a0627ac-d50a-4442-a0c5-83157f3470a6-kube-api-access-bdr8q\") pod \"test-curl-7839-ab9061\" (UID: \"0a0627ac-d50a-4442-a0c5-83157f3470a6\") " pod="opendatahub/test-curl-7839-ab9061"
Sep 11 18:30:12.148507 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.148471    2576 generic.go:356] "Generic (PLEG): container finished" podID="3ac93d8a-158d-42b3-8b85-3217f8fab901" containerID="27e1c4d5299122e1bb748fd5a73f1f896a349e9c6a345c59dfe7f4cd5a24824d" exitCode=0
Sep 11 18:30:12.148967 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.148513    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2"
Sep 11 18:30:12.148967 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.148559    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2" event={"ID":"3ac93d8a-158d-42b3-8b85-3217f8fab901","Type":"ContainerDied","Data":"27e1c4d5299122e1bb748fd5a73f1f896a349e9c6a345c59dfe7f4cd5a24824d"}
Sep 11 18:30:12.148967 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.148629    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2" event={"ID":"3ac93d8a-158d-42b3-8b85-3217f8fab901","Type":"ContainerDied","Data":"c03ab51f42d853c56412e6dfb66f5f73e6e5af432a8c3376447aae7437f3b314"}
Sep 11 18:30:12.148967 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.148658    2576 scope.go:117] "RemoveContainer" containerID="27e1c4d5299122e1bb748fd5a73f1f896a349e9c6a345c59dfe7f4cd5a24824d"
Sep 11 18:30:12.150042 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.150013    2576 generic.go:356] "Generic (PLEG): container finished" podID="8bf1b9de-9a29-4fbf-9ace-ea260daabbf8" containerID="5f57224806651c5e095de3d1ac54341ad707adc98d581a9de30cc289b0d847d4" exitCode=0
Sep 11 18:30:12.150226 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.150045    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6" event={"ID":"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8","Type":"ContainerDied","Data":"5f57224806651c5e095de3d1ac54341ad707adc98d581a9de30cc289b0d847d4"}
Sep 11 18:30:12.150226 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.150092    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6" event={"ID":"8bf1b9de-9a29-4fbf-9ace-ea260daabbf8","Type":"ContainerDied","Data":"3111fa85ef4898062e98d7826a0373af7dd25610a2a0b6f2d940621362179b4e"}
Sep 11 18:30:12.150226 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.150054    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"
Sep 11 18:30:12.150437 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:30:12.150415    2576 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7839-87d585" container="test-curl-7839-87d585"
Sep 11 18:30:12.161036 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.161017    2576 scope.go:117] "RemoveContainer" containerID="27e1c4d5299122e1bb748fd5a73f1f896a349e9c6a345c59dfe7f4cd5a24824d"
Sep 11 18:30:12.161325 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:30:12.161296    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27e1c4d5299122e1bb748fd5a73f1f896a349e9c6a345c59dfe7f4cd5a24824d\": container with ID starting with 27e1c4d5299122e1bb748fd5a73f1f896a349e9c6a345c59dfe7f4cd5a24824d not found: ID does not exist" containerID="27e1c4d5299122e1bb748fd5a73f1f896a349e9c6a345c59dfe7f4cd5a24824d"
Sep 11 18:30:12.161423 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.161334    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e1c4d5299122e1bb748fd5a73f1f896a349e9c6a345c59dfe7f4cd5a24824d"} err="failed to get container status \"27e1c4d5299122e1bb748fd5a73f1f896a349e9c6a345c59dfe7f4cd5a24824d\": rpc error: code = NotFound desc = could not find container \"27e1c4d5299122e1bb748fd5a73f1f896a349e9c6a345c59dfe7f4cd5a24824d\": container with ID starting with 27e1c4d5299122e1bb748fd5a73f1f896a349e9c6a345c59dfe7f4cd5a24824d not found: ID does not exist"
Sep 11 18:30:12.161423 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.161361    2576 scope.go:117] "RemoveContainer" containerID="5f57224806651c5e095de3d1ac54341ad707adc98d581a9de30cc289b0d847d4"
Sep 11 18:30:12.169927 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.169904    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7839-ab9061"
Sep 11 18:30:12.172050 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.172029    2576 scope.go:117] "RemoveContainer" containerID="5f57224806651c5e095de3d1ac54341ad707adc98d581a9de30cc289b0d847d4"
Sep 11 18:30:12.172370 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:30:12.172344    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f57224806651c5e095de3d1ac54341ad707adc98d581a9de30cc289b0d847d4\": container with ID starting with 5f57224806651c5e095de3d1ac54341ad707adc98d581a9de30cc289b0d847d4 not found: ID does not exist" containerID="5f57224806651c5e095de3d1ac54341ad707adc98d581a9de30cc289b0d847d4"
Sep 11 18:30:12.172462 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.172375    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f57224806651c5e095de3d1ac54341ad707adc98d581a9de30cc289b0d847d4"} err="failed to get container status \"5f57224806651c5e095de3d1ac54341ad707adc98d581a9de30cc289b0d847d4\": rpc error: code = NotFound desc = could not find container \"5f57224806651c5e095de3d1ac54341ad707adc98d581a9de30cc289b0d847d4\": container with ID starting with 5f57224806651c5e095de3d1ac54341ad707adc98d581a9de30cc289b0d847d4 not found: ID does not exist"
Sep 11 18:30:12.175641 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.175621    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"]
Sep 11 18:30:12.180459 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.180434    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-12eb53-68bc6f96-krsd6"]
Sep 11 18:30:12.194018 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.193987    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2"]
Sep 11 18:30:12.195923 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.195901    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-12eb53-577fc5b8c4-86hb2"]
Sep 11 18:30:12.442576 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:12.442546    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7839-ab9061"]
Sep 11 18:30:12.445166 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:30:12.445140    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a0627ac_d50a_4442_a0c5_83157f3470a6.slice/crio-4d4886ffb6de19dea2d37bf54cd742ebff6de31e34d1a2bceccbc3b2a90d25c6 WatchSource:0}: Error finding container 4d4886ffb6de19dea2d37bf54cd742ebff6de31e34d1a2bceccbc3b2a90d25c6: Status 404 returned error can't find the container with id 4d4886ffb6de19dea2d37bf54cd742ebff6de31e34d1a2bceccbc3b2a90d25c6
Sep 11 18:30:13.158757 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.158727    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7839-ab9061" event={"ID":"0a0627ac-d50a-4442-a0c5-83157f3470a6","Type":"ContainerStarted","Data":"326d163b000b06418d4c5b04b636f6d7c6ac659b55cc21821e9e943a26ccde1f"}
Sep 11 18:30:13.159092 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.158773    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7839-ab9061" event={"ID":"0a0627ac-d50a-4442-a0c5-83157f3470a6","Type":"ContainerStarted","Data":"4d4886ffb6de19dea2d37bf54cd742ebff6de31e34d1a2bceccbc3b2a90d25c6"}
Sep 11 18:30:13.159092 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.158849    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7839-87d585" podUID="584b719a-d733-40d4-a6f4-5b4f86080f56" containerName="test-curl-7839-87d585" containerID="cri-o://00cfb22445865e63c910c2b7fce1d7ba194d1a9e43098f9bc3488d363e87e44b" gracePeriod=2
Sep 11 18:30:13.176303 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.176190    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7839-ab9061" podStartSLOduration=2.176168925 podStartE2EDuration="2.176168925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:13.175898365 +0000 UTC m=+1211.777259526" watchObservedRunningTime="2026-09-11 18:30:13.176168925 +0000 UTC m=+1211.777530090"
Sep 11 18:30:13.337224 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.337195    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7839-3b6d08"
Sep 11 18:30:13.424435 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.424344    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zx94\" (UniqueName: \"kubernetes.io/projected/aa7af295-5505-4a32-927e-94d5928802cf-kube-api-access-4zx94\") pod \"aa7af295-5505-4a32-927e-94d5928802cf\" (UID: \"aa7af295-5505-4a32-927e-94d5928802cf\") "
Sep 11 18:30:13.426448 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.426421    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7af295-5505-4a32-927e-94d5928802cf-kube-api-access-4zx94" (OuterVolumeSpecName: "kube-api-access-4zx94") pod "aa7af295-5505-4a32-927e-94d5928802cf" (UID: "aa7af295-5505-4a32-927e-94d5928802cf"). InnerVolumeSpecName "kube-api-access-4zx94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:30:13.524988 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.524955    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zx94\" (UniqueName: \"kubernetes.io/projected/aa7af295-5505-4a32-927e-94d5928802cf-kube-api-access-4zx94\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:30:13.721463 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.721423    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7839-ab9061"]
Sep 11 18:30:13.727370 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.727344    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7839-ab9061"]
Sep 11 18:30:13.844721 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.844682    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7839-962ad1"]
Sep 11 18:30:13.845101 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.845085    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf1b9de-9a29-4fbf-9ace-ea260daabbf8" containerName="maas-api"
Sep 11 18:30:13.845138 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.845104    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf1b9de-9a29-4fbf-9ace-ea260daabbf8" containerName="maas-api"
Sep 11 18:30:13.845138 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.845116    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a0627ac-d50a-4442-a0c5-83157f3470a6" containerName="test-curl-7839-ab9061"
Sep 11 18:30:13.845138 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.845122    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a0627ac-d50a-4442-a0c5-83157f3470a6" containerName="test-curl-7839-ab9061"
Sep 11 18:30:13.845138 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.845136    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ac93d8a-158d-42b3-8b85-3217f8fab901" containerName="payload-pre-processing"
Sep 11 18:30:13.845281 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.845142    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ac93d8a-158d-42b3-8b85-3217f8fab901" containerName="payload-pre-processing"
Sep 11 18:30:13.845281 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.845222    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ac93d8a-158d-42b3-8b85-3217f8fab901" containerName="payload-pre-processing"
Sep 11 18:30:13.845281 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.845231    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf1b9de-9a29-4fbf-9ace-ea260daabbf8" containerName="maas-api"
Sep 11 18:30:13.845281 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.845240    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a0627ac-d50a-4442-a0c5-83157f3470a6" containerName="test-curl-7839-ab9061"
Sep 11 18:30:13.848208 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.848191    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7839-962ad1"
Sep 11 18:30:13.856553 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.856525    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7839-962ad1"]
Sep 11 18:30:13.910405 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.910374    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ac93d8a-158d-42b3-8b85-3217f8fab901" path="/var/lib/kubelet/pods/3ac93d8a-158d-42b3-8b85-3217f8fab901/volumes"
Sep 11 18:30:13.910800 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.910786    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bf1b9de-9a29-4fbf-9ace-ea260daabbf8" path="/var/lib/kubelet/pods/8bf1b9de-9a29-4fbf-9ace-ea260daabbf8/volumes"
Sep 11 18:30:13.911070 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.911058    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa7af295-5505-4a32-927e-94d5928802cf" path="/var/lib/kubelet/pods/aa7af295-5505-4a32-927e-94d5928802cf/volumes"
Sep 11 18:30:13.928376 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:13.928339    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cknwg\" (UniqueName: \"kubernetes.io/projected/a512cc58-5299-459a-8104-2fd020b69029-kube-api-access-cknwg\") pod \"test-curl-7839-962ad1\" (UID: \"a512cc58-5299-459a-8104-2fd020b69029\") " pod="opendatahub/test-curl-7839-962ad1"
Sep 11 18:30:14.029660 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.029555    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cknwg\" (UniqueName: \"kubernetes.io/projected/a512cc58-5299-459a-8104-2fd020b69029-kube-api-access-cknwg\") pod \"test-curl-7839-962ad1\" (UID: \"a512cc58-5299-459a-8104-2fd020b69029\") " pod="opendatahub/test-curl-7839-962ad1"
Sep 11 18:30:14.037455 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.037419    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cknwg\" (UniqueName: \"kubernetes.io/projected/a512cc58-5299-459a-8104-2fd020b69029-kube-api-access-cknwg\") pod \"test-curl-7839-962ad1\" (UID: \"a512cc58-5299-459a-8104-2fd020b69029\") " pod="opendatahub/test-curl-7839-962ad1"
Sep 11 18:30:14.086990 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.086957    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"]
Sep 11 18:30:14.091374 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.091357    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:30:14.093537 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.093517    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-553596\"/\"e2e-conflict-b-c17caa91-kserve-self-signed-certs\""
Sep 11 18:30:14.098922 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.098900    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"]
Sep 11 18:30:14.157416 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.157377    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7839-962ad1"
Sep 11 18:30:14.167449 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.167417    2576 generic.go:356] "Generic (PLEG): container finished" podID="aa7af295-5505-4a32-927e-94d5928802cf" containerID="55c0c541f77abacf809a23fc17276a82dfb92f45e9102e45e42fdd7bcc531bc0" exitCode=137
Sep 11 18:30:14.167881 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.167484    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7839-3b6d08"
Sep 11 18:30:14.167881 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.167529    2576 scope.go:117] "RemoveContainer" containerID="55c0c541f77abacf809a23fc17276a82dfb92f45e9102e45e42fdd7bcc531bc0"
Sep 11 18:30:14.167881 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:30:14.167835    2576 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7839-ab9061" container="test-curl-7839-ab9061"
Sep 11 18:30:14.188372 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.188351    2576 scope.go:117] "RemoveContainer" containerID="55c0c541f77abacf809a23fc17276a82dfb92f45e9102e45e42fdd7bcc531bc0"
Sep 11 18:30:14.188811 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:30:14.188787    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55c0c541f77abacf809a23fc17276a82dfb92f45e9102e45e42fdd7bcc531bc0\": container with ID starting with 55c0c541f77abacf809a23fc17276a82dfb92f45e9102e45e42fdd7bcc531bc0 not found: ID does not exist" containerID="55c0c541f77abacf809a23fc17276a82dfb92f45e9102e45e42fdd7bcc531bc0"
Sep 11 18:30:14.188920 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.188819    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55c0c541f77abacf809a23fc17276a82dfb92f45e9102e45e42fdd7bcc531bc0"} err="failed to get container status \"55c0c541f77abacf809a23fc17276a82dfb92f45e9102e45e42fdd7bcc531bc0\": rpc error: code = NotFound desc = could not find container \"55c0c541f77abacf809a23fc17276a82dfb92f45e9102e45e42fdd7bcc531bc0\": container with ID starting with 55c0c541f77abacf809a23fc17276a82dfb92f45e9102e45e42fdd7bcc531bc0 not found: ID does not exist"
Sep 11 18:30:14.231188 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.231156    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e373bbfd-9aab-4552-83d1-6dfdd4685563-tmp-dir\") pod \"e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd\" (UID: \"e373bbfd-9aab-4552-83d1-6dfdd4685563\") " pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:30:14.231341 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.231195    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e373bbfd-9aab-4552-83d1-6dfdd4685563-home\") pod \"e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd\" (UID: \"e373bbfd-9aab-4552-83d1-6dfdd4685563\") " pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:30:14.231341 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.231216    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e373bbfd-9aab-4552-83d1-6dfdd4685563-model-cache\") pod \"e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd\" (UID: \"e373bbfd-9aab-4552-83d1-6dfdd4685563\") " pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:30:14.231341 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.231231    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e373bbfd-9aab-4552-83d1-6dfdd4685563-tls-certs\") pod \"e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd\" (UID: \"e373bbfd-9aab-4552-83d1-6dfdd4685563\") " pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:30:14.231341 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.231334    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e373bbfd-9aab-4552-83d1-6dfdd4685563-dshm\") pod \"e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd\" (UID: \"e373bbfd-9aab-4552-83d1-6dfdd4685563\") " pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:30:14.231492 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.231371    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt7mj\" (UniqueName: \"kubernetes.io/projected/e373bbfd-9aab-4552-83d1-6dfdd4685563-kube-api-access-zt7mj\") pod \"e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd\" (UID: \"e373bbfd-9aab-4552-83d1-6dfdd4685563\") " pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:30:14.332743 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.332648    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt7mj\" (UniqueName: \"kubernetes.io/projected/e373bbfd-9aab-4552-83d1-6dfdd4685563-kube-api-access-zt7mj\") pod \"e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd\" (UID: \"e373bbfd-9aab-4552-83d1-6dfdd4685563\") " pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:30:14.332898 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.332817    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e373bbfd-9aab-4552-83d1-6dfdd4685563-tmp-dir\") pod \"e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd\" (UID: \"e373bbfd-9aab-4552-83d1-6dfdd4685563\") " pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:30:14.332898 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.332868    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e373bbfd-9aab-4552-83d1-6dfdd4685563-home\") pod \"e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd\" (UID: \"e373bbfd-9aab-4552-83d1-6dfdd4685563\") " pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:30:14.332978 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.332905    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e373bbfd-9aab-4552-83d1-6dfdd4685563-model-cache\") pod \"e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd\" (UID: \"e373bbfd-9aab-4552-83d1-6dfdd4685563\") " pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:30:14.332978 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.332927    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e373bbfd-9aab-4552-83d1-6dfdd4685563-tls-certs\") pod \"e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd\" (UID: \"e373bbfd-9aab-4552-83d1-6dfdd4685563\") " pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:30:14.333053 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.332983    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e373bbfd-9aab-4552-83d1-6dfdd4685563-dshm\") pod \"e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd\" (UID: \"e373bbfd-9aab-4552-83d1-6dfdd4685563\") " pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:30:14.333292 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.333270    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e373bbfd-9aab-4552-83d1-6dfdd4685563-tmp-dir\") pod \"e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd\" (UID: \"e373bbfd-9aab-4552-83d1-6dfdd4685563\") " pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:30:14.333341 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.333287    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e373bbfd-9aab-4552-83d1-6dfdd4685563-model-cache\") pod \"e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd\" (UID: \"e373bbfd-9aab-4552-83d1-6dfdd4685563\") " pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:30:14.333377 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.333338    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e373bbfd-9aab-4552-83d1-6dfdd4685563-home\") pod \"e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd\" (UID: \"e373bbfd-9aab-4552-83d1-6dfdd4685563\") " pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:30:14.335228 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.335195    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e373bbfd-9aab-4552-83d1-6dfdd4685563-dshm\") pod \"e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd\" (UID: \"e373bbfd-9aab-4552-83d1-6dfdd4685563\") " pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:30:14.335446 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.335424    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e373bbfd-9aab-4552-83d1-6dfdd4685563-tls-certs\") pod \"e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd\" (UID: \"e373bbfd-9aab-4552-83d1-6dfdd4685563\") " pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:30:14.340260 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.340229    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt7mj\" (UniqueName: \"kubernetes.io/projected/e373bbfd-9aab-4552-83d1-6dfdd4685563-kube-api-access-zt7mj\") pod \"e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd\" (UID: \"e373bbfd-9aab-4552-83d1-6dfdd4685563\") " pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:30:14.401052 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.401011    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:30:14.435269 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.435235    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7839-962ad1"]
Sep 11 18:30:14.436251 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:30:14.436224    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda512cc58_5299_459a_8104_2fd020b69029.slice/crio-aff8420c696f71628a5654a886e2aed17ae66ea47ab11498e79ff202ec314265 WatchSource:0}: Error finding container aff8420c696f71628a5654a886e2aed17ae66ea47ab11498e79ff202ec314265: Status 404 returned error can't find the container with id aff8420c696f71628a5654a886e2aed17ae66ea47ab11498e79ff202ec314265
Sep 11 18:30:14.682327 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:14.682278    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"]
Sep 11 18:30:14.684529 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:30:14.684504    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode373bbfd_9aab_4552_83d1_6dfdd4685563.slice/crio-1315a055751b4118fbe359564aa7cb47dace9f9758f585320607c3ae57791ed6 WatchSource:0}: Error finding container 1315a055751b4118fbe359564aa7cb47dace9f9758f585320607c3ae57791ed6: Status 404 returned error can't find the container with id 1315a055751b4118fbe359564aa7cb47dace9f9758f585320607c3ae57791ed6
Sep 11 18:30:15.174392 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:15.174357    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd" event={"ID":"e373bbfd-9aab-4552-83d1-6dfdd4685563","Type":"ContainerStarted","Data":"e54b704511db34af82f281d97d9262a19430ad4ae554a8268adcaad191e12367"}
Sep 11 18:30:15.174767 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:15.174405    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd" event={"ID":"e373bbfd-9aab-4552-83d1-6dfdd4685563","Type":"ContainerStarted","Data":"1315a055751b4118fbe359564aa7cb47dace9f9758f585320607c3ae57791ed6"}
Sep 11 18:30:15.175953 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:15.175927    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7839-962ad1" event={"ID":"a512cc58-5299-459a-8104-2fd020b69029","Type":"ContainerStarted","Data":"38c6befa478838324fdee81d7fdf7d3420ad2cfd612f62eed9e597ccdf873f6c"}
Sep 11 18:30:15.176047 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:15.175962    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7839-962ad1" event={"ID":"a512cc58-5299-459a-8104-2fd020b69029","Type":"ContainerStarted","Data":"aff8420c696f71628a5654a886e2aed17ae66ea47ab11498e79ff202ec314265"}
Sep 11 18:30:15.176886 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:15.176863    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7839-ab9061" podUID="0a0627ac-d50a-4442-a0c5-83157f3470a6" containerName="test-curl-7839-ab9061" containerID="cri-o://326d163b000b06418d4c5b04b636f6d7c6ac659b55cc21821e9e943a26ccde1f" gracePeriod=2
Sep 11 18:30:15.193603 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:15.193532    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd" podStartSLOduration=1.193512791 podStartE2EDuration="1.193512791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:15.191522082 +0000 UTC m=+1213.792883247" watchObservedRunningTime="2026-09-11 18:30:15.193512791 +0000 UTC m=+1213.794873954"
Sep 11 18:30:15.208463 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:15.208414    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7839-962ad1" podStartSLOduration=2.208395868 podStartE2EDuration="2.208395868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:15.207620373 +0000 UTC m=+1213.808981538" watchObservedRunningTime="2026-09-11 18:30:15.208395868 +0000 UTC m=+1213.809757032"
Sep 11 18:30:15.346482 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:15.346461    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7839-87d585"
Sep 11 18:30:15.443554 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:15.443529    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zd87t\" (UniqueName: \"kubernetes.io/projected/584b719a-d733-40d4-a6f4-5b4f86080f56-kube-api-access-zd87t\") pod \"584b719a-d733-40d4-a6f4-5b4f86080f56\" (UID: \"584b719a-d733-40d4-a6f4-5b4f86080f56\") "
Sep 11 18:30:15.445565 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:15.445536    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/584b719a-d733-40d4-a6f4-5b4f86080f56-kube-api-access-zd87t" (OuterVolumeSpecName: "kube-api-access-zd87t") pod "584b719a-d733-40d4-a6f4-5b4f86080f56" (UID: "584b719a-d733-40d4-a6f4-5b4f86080f56"). InnerVolumeSpecName "kube-api-access-zd87t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:30:15.545151 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:15.545116    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd87t\" (UniqueName: \"kubernetes.io/projected/584b719a-d733-40d4-a6f4-5b4f86080f56-kube-api-access-zd87t\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:30:15.757343 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:15.757258    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7839-962ad1"]
Sep 11 18:30:15.762964 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:15.762936    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7839-962ad1"]
Sep 11 18:30:15.910220 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:15.910187    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="584b719a-d733-40d4-a6f4-5b4f86080f56" path="/var/lib/kubelet/pods/584b719a-d733-40d4-a6f4-5b4f86080f56/volumes"
Sep 11 18:30:16.186405 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:16.186299    2576 generic.go:356] "Generic (PLEG): container finished" podID="584b719a-d733-40d4-a6f4-5b4f86080f56" containerID="00cfb22445865e63c910c2b7fce1d7ba194d1a9e43098f9bc3488d363e87e44b" exitCode=137
Sep 11 18:30:16.186405 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:16.186341    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7839-87d585"
Sep 11 18:30:16.186405 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:16.186399    2576 scope.go:117] "RemoveContainer" containerID="00cfb22445865e63c910c2b7fce1d7ba194d1a9e43098f9bc3488d363e87e44b"
Sep 11 18:30:16.186974 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:30:16.186706    2576 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7839-962ad1" container="test-curl-7839-962ad1"
Sep 11 18:30:16.194759 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:16.194739    2576 scope.go:117] "RemoveContainer" containerID="00cfb22445865e63c910c2b7fce1d7ba194d1a9e43098f9bc3488d363e87e44b"
Sep 11 18:30:16.195057 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:30:16.195040    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00cfb22445865e63c910c2b7fce1d7ba194d1a9e43098f9bc3488d363e87e44b\": container with ID starting with 00cfb22445865e63c910c2b7fce1d7ba194d1a9e43098f9bc3488d363e87e44b not found: ID does not exist" containerID="00cfb22445865e63c910c2b7fce1d7ba194d1a9e43098f9bc3488d363e87e44b"
Sep 11 18:30:16.195129 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:16.195064    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00cfb22445865e63c910c2b7fce1d7ba194d1a9e43098f9bc3488d363e87e44b"} err="failed to get container status \"00cfb22445865e63c910c2b7fce1d7ba194d1a9e43098f9bc3488d363e87e44b\": rpc error: code = NotFound desc = could not find container \"00cfb22445865e63c910c2b7fce1d7ba194d1a9e43098f9bc3488d363e87e44b\": container with ID starting with 00cfb22445865e63c910c2b7fce1d7ba194d1a9e43098f9bc3488d363e87e44b not found: ID does not exist"
Sep 11 18:30:17.192861 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:17.192829    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7839-962ad1" podUID="a512cc58-5299-459a-8104-2fd020b69029" containerName="test-curl-7839-962ad1" containerID="cri-o://38c6befa478838324fdee81d7fdf7d3420ad2cfd612f62eed9e597ccdf873f6c" gracePeriod=2
Sep 11 18:30:17.369547 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:17.369517    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7839-ab9061"
Sep 11 18:30:17.462493 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:17.462462    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdr8q\" (UniqueName: \"kubernetes.io/projected/0a0627ac-d50a-4442-a0c5-83157f3470a6-kube-api-access-bdr8q\") pod \"0a0627ac-d50a-4442-a0c5-83157f3470a6\" (UID: \"0a0627ac-d50a-4442-a0c5-83157f3470a6\") "
Sep 11 18:30:17.464541 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:17.464517    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0627ac-d50a-4442-a0c5-83157f3470a6-kube-api-access-bdr8q" (OuterVolumeSpecName: "kube-api-access-bdr8q") pod "0a0627ac-d50a-4442-a0c5-83157f3470a6" (UID: "0a0627ac-d50a-4442-a0c5-83157f3470a6"). InnerVolumeSpecName "kube-api-access-bdr8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:30:17.563308 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:17.563273    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdr8q\" (UniqueName: \"kubernetes.io/projected/0a0627ac-d50a-4442-a0c5-83157f3470a6-kube-api-access-bdr8q\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:30:17.910420 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:17.910325    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a0627ac-d50a-4442-a0c5-83157f3470a6" path="/var/lib/kubelet/pods/0a0627ac-d50a-4442-a0c5-83157f3470a6/volumes"
Sep 11 18:30:18.198497 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:18.198463    2576 generic.go:356] "Generic (PLEG): container finished" podID="0a0627ac-d50a-4442-a0c5-83157f3470a6" containerID="326d163b000b06418d4c5b04b636f6d7c6ac659b55cc21821e9e943a26ccde1f" exitCode=137
Sep 11 18:30:18.198914 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:18.198504    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7839-ab9061"
Sep 11 18:30:18.198914 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:18.198562    2576 scope.go:117] "RemoveContainer" containerID="326d163b000b06418d4c5b04b636f6d7c6ac659b55cc21821e9e943a26ccde1f"
Sep 11 18:30:18.206827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:18.206800    2576 scope.go:117] "RemoveContainer" containerID="326d163b000b06418d4c5b04b636f6d7c6ac659b55cc21821e9e943a26ccde1f"
Sep 11 18:30:18.207108 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:30:18.207089    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"326d163b000b06418d4c5b04b636f6d7c6ac659b55cc21821e9e943a26ccde1f\": container with ID starting with 326d163b000b06418d4c5b04b636f6d7c6ac659b55cc21821e9e943a26ccde1f not found: ID does not exist" containerID="326d163b000b06418d4c5b04b636f6d7c6ac659b55cc21821e9e943a26ccde1f"
Sep 11 18:30:18.207167 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:18.207120    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"326d163b000b06418d4c5b04b636f6d7c6ac659b55cc21821e9e943a26ccde1f"} err="failed to get container status \"326d163b000b06418d4c5b04b636f6d7c6ac659b55cc21821e9e943a26ccde1f\": rpc error: code = NotFound desc = could not find container \"326d163b000b06418d4c5b04b636f6d7c6ac659b55cc21821e9e943a26ccde1f\": container with ID starting with 326d163b000b06418d4c5b04b636f6d7c6ac659b55cc21821e9e943a26ccde1f not found: ID does not exist"
Sep 11 18:30:19.360371 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:19.360348    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7839-962ad1"
Sep 11 18:30:19.480548 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:19.480449    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cknwg\" (UniqueName: \"kubernetes.io/projected/a512cc58-5299-459a-8104-2fd020b69029-kube-api-access-cknwg\") pod \"a512cc58-5299-459a-8104-2fd020b69029\" (UID: \"a512cc58-5299-459a-8104-2fd020b69029\") "
Sep 11 18:30:19.482618 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:19.482574    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a512cc58-5299-459a-8104-2fd020b69029-kube-api-access-cknwg" (OuterVolumeSpecName: "kube-api-access-cknwg") pod "a512cc58-5299-459a-8104-2fd020b69029" (UID: "a512cc58-5299-459a-8104-2fd020b69029"). InnerVolumeSpecName "kube-api-access-cknwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:30:19.581602 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:19.581536    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cknwg\" (UniqueName: \"kubernetes.io/projected/a512cc58-5299-459a-8104-2fd020b69029-kube-api-access-cknwg\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:30:19.910185 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:19.910088    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a512cc58-5299-459a-8104-2fd020b69029" path="/var/lib/kubelet/pods/a512cc58-5299-459a-8104-2fd020b69029/volumes"
Sep 11 18:30:20.210945 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:20.210913    2576 generic.go:356] "Generic (PLEG): container finished" podID="a512cc58-5299-459a-8104-2fd020b69029" containerID="38c6befa478838324fdee81d7fdf7d3420ad2cfd612f62eed9e597ccdf873f6c" exitCode=137
Sep 11 18:30:20.211156 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:20.210973    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7839-962ad1"
Sep 11 18:30:20.211156 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:20.211015    2576 scope.go:117] "RemoveContainer" containerID="38c6befa478838324fdee81d7fdf7d3420ad2cfd612f62eed9e597ccdf873f6c"
Sep 11 18:30:20.219387 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:20.219372    2576 scope.go:117] "RemoveContainer" containerID="38c6befa478838324fdee81d7fdf7d3420ad2cfd612f62eed9e597ccdf873f6c"
Sep 11 18:30:20.219664 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:30:20.219646    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38c6befa478838324fdee81d7fdf7d3420ad2cfd612f62eed9e597ccdf873f6c\": container with ID starting with 38c6befa478838324fdee81d7fdf7d3420ad2cfd612f62eed9e597ccdf873f6c not found: ID does not exist" containerID="38c6befa478838324fdee81d7fdf7d3420ad2cfd612f62eed9e597ccdf873f6c"
Sep 11 18:30:20.219722 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:20.219671    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38c6befa478838324fdee81d7fdf7d3420ad2cfd612f62eed9e597ccdf873f6c"} err="failed to get container status \"38c6befa478838324fdee81d7fdf7d3420ad2cfd612f62eed9e597ccdf873f6c\": rpc error: code = NotFound desc = could not find container \"38c6befa478838324fdee81d7fdf7d3420ad2cfd612f62eed9e597ccdf873f6c\": container with ID starting with 38c6befa478838324fdee81d7fdf7d3420ad2cfd612f62eed9e597ccdf873f6c not found: ID does not exist"
Sep 11 18:30:23.611967 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:23.611930    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:30:24.401472 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:24.401435    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:30:24.401673 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:24.401504    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:30:24.413888 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:24.413863    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:30:25.073728 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.073689    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4"]
Sep 11 18:30:25.074088 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.074063    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a512cc58-5299-459a-8104-2fd020b69029" containerName="test-curl-7839-962ad1"
Sep 11 18:30:25.074088 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.074075    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a512cc58-5299-459a-8104-2fd020b69029" containerName="test-curl-7839-962ad1"
Sep 11 18:30:25.074178 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.074130    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a512cc58-5299-459a-8104-2fd020b69029" containerName="test-curl-7839-962ad1"
Sep 11 18:30:25.078274 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.078257    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4"
Sep 11 18:30:25.080570 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.080548    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3e24a782-dockercfg-nqzxm\""
Sep 11 18:30:25.080742 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.080599    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3e24a782\""
Sep 11 18:30:25.086759 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.086734    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4"]
Sep 11 18:30:25.134535 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.134503    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8"]
Sep 11 18:30:25.144228 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.144198    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8"
Sep 11 18:30:25.156251 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.156221    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8"]
Sep 11 18:30:25.242402 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.242370    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrltl\" (UniqueName: \"kubernetes.io/projected/281d75e1-25ac-4c52-bad4-14f88222061e-kube-api-access-jrltl\") pod \"payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4\" (UID: \"281d75e1-25ac-4c52-bad4-14f88222061e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4"
Sep 11 18:30:25.242572 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.242414    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281d75e1-25ac-4c52-bad4-14f88222061e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4\" (UID: \"281d75e1-25ac-4c52-bad4-14f88222061e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4"
Sep 11 18:30:25.246796 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.246767    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:30:25.343714 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.343625    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078cd6e6-514e-458d-840e-415c21f14bc8-plugins-config-volume\") pod \"payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8\" (UID: \"078cd6e6-514e-458d-840e-415c21f14bc8\") " pod="openshift-ingress/payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8"
Sep 11 18:30:25.343891 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.343743    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrltl\" (UniqueName: \"kubernetes.io/projected/281d75e1-25ac-4c52-bad4-14f88222061e-kube-api-access-jrltl\") pod \"payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4\" (UID: \"281d75e1-25ac-4c52-bad4-14f88222061e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4"
Sep 11 18:30:25.343891 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.343798    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzpxl\" (UniqueName: \"kubernetes.io/projected/078cd6e6-514e-458d-840e-415c21f14bc8-kube-api-access-wzpxl\") pod \"payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8\" (UID: \"078cd6e6-514e-458d-840e-415c21f14bc8\") " pod="openshift-ingress/payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8"
Sep 11 18:30:25.343891 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.343831    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281d75e1-25ac-4c52-bad4-14f88222061e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4\" (UID: \"281d75e1-25ac-4c52-bad4-14f88222061e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4"
Sep 11 18:30:25.344716 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.344698    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281d75e1-25ac-4c52-bad4-14f88222061e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4\" (UID: \"281d75e1-25ac-4c52-bad4-14f88222061e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4"
Sep 11 18:30:25.354835 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.354802    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrltl\" (UniqueName: \"kubernetes.io/projected/281d75e1-25ac-4c52-bad4-14f88222061e-kube-api-access-jrltl\") pod \"payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4\" (UID: \"281d75e1-25ac-4c52-bad4-14f88222061e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4"
Sep 11 18:30:25.389436 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.389402    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4"
Sep 11 18:30:25.445189 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.445158    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078cd6e6-514e-458d-840e-415c21f14bc8-plugins-config-volume\") pod \"payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8\" (UID: \"078cd6e6-514e-458d-840e-415c21f14bc8\") " pod="openshift-ingress/payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8"
Sep 11 18:30:25.445324 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.445217    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzpxl\" (UniqueName: \"kubernetes.io/projected/078cd6e6-514e-458d-840e-415c21f14bc8-kube-api-access-wzpxl\") pod \"payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8\" (UID: \"078cd6e6-514e-458d-840e-415c21f14bc8\") " pod="openshift-ingress/payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8"
Sep 11 18:30:25.446218 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.446191    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078cd6e6-514e-458d-840e-415c21f14bc8-plugins-config-volume\") pod \"payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8\" (UID: \"078cd6e6-514e-458d-840e-415c21f14bc8\") " pod="openshift-ingress/payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8"
Sep 11 18:30:25.454780 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.454729    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzpxl\" (UniqueName: \"kubernetes.io/projected/078cd6e6-514e-458d-840e-415c21f14bc8-kube-api-access-wzpxl\") pod \"payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8\" (UID: \"078cd6e6-514e-458d-840e-415c21f14bc8\") " pod="openshift-ingress/payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8"
Sep 11 18:30:25.665775 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.665729    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4"]
Sep 11 18:30:25.668174 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:30:25.668147    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod281d75e1_25ac_4c52_bad4_14f88222061e.slice/crio-09074c67edf5936aa00004ef101eeda889a0198b112785f90e705fcd9922e6b0 WatchSource:0}: Error finding container 09074c67edf5936aa00004ef101eeda889a0198b112785f90e705fcd9922e6b0: Status 404 returned error can't find the container with id 09074c67edf5936aa00004ef101eeda889a0198b112785f90e705fcd9922e6b0
Sep 11 18:30:25.753862 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.753834    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8"
Sep 11 18:30:25.940681 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.940646    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx"]
Sep 11 18:30:25.945878 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.945858    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx"
Sep 11 18:30:25.948285 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.948261    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-bf9898-dockercfg-czg6k\""
Sep 11 18:30:25.948285 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.948270    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-bf9898\""
Sep 11 18:30:25.950229 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.950205    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e4fb425-8f69-41f9-8eab-33a44f90a28a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx\" (UID: \"1e4fb425-8f69-41f9-8eab-33a44f90a28a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx"
Sep 11 18:30:25.950322 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.950249    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52m4c\" (UniqueName: \"kubernetes.io/projected/1e4fb425-8f69-41f9-8eab-33a44f90a28a-kube-api-access-52m4c\") pod \"payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx\" (UID: \"1e4fb425-8f69-41f9-8eab-33a44f90a28a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx"
Sep 11 18:30:25.958757 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:25.958735    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx"]
Sep 11 18:30:26.044154 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:26.044120    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8"]
Sep 11 18:30:26.045414 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:30:26.045380    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078cd6e6_514e_458d_840e_415c21f14bc8.slice/crio-7424c31a9cefe458030f805dc9eca06a1ddac3f088e85f854adacf4afe5bd045 WatchSource:0}: Error finding container 7424c31a9cefe458030f805dc9eca06a1ddac3f088e85f854adacf4afe5bd045: Status 404 returned error can't find the container with id 7424c31a9cefe458030f805dc9eca06a1ddac3f088e85f854adacf4afe5bd045
Sep 11 18:30:26.050638 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:26.050615    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e4fb425-8f69-41f9-8eab-33a44f90a28a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx\" (UID: \"1e4fb425-8f69-41f9-8eab-33a44f90a28a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx"
Sep 11 18:30:26.050720 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:26.050653    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52m4c\" (UniqueName: \"kubernetes.io/projected/1e4fb425-8f69-41f9-8eab-33a44f90a28a-kube-api-access-52m4c\") pod \"payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx\" (UID: \"1e4fb425-8f69-41f9-8eab-33a44f90a28a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx"
Sep 11 18:30:26.051268 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:26.051251    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e4fb425-8f69-41f9-8eab-33a44f90a28a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx\" (UID: \"1e4fb425-8f69-41f9-8eab-33a44f90a28a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx"
Sep 11 18:30:26.061698 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:26.061674    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52m4c\" (UniqueName: \"kubernetes.io/projected/1e4fb425-8f69-41f9-8eab-33a44f90a28a-kube-api-access-52m4c\") pod \"payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx\" (UID: \"1e4fb425-8f69-41f9-8eab-33a44f90a28a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx"
Sep 11 18:30:26.245383 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:26.245275    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8" event={"ID":"078cd6e6-514e-458d-840e-415c21f14bc8","Type":"ContainerStarted","Data":"fdb01ca240939349e432b9f254e2f977bf41ca7a997dabd2743059ad6b7200d8"}
Sep 11 18:30:26.245383 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:26.245335    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8" event={"ID":"078cd6e6-514e-458d-840e-415c21f14bc8","Type":"ContainerStarted","Data":"7424c31a9cefe458030f805dc9eca06a1ddac3f088e85f854adacf4afe5bd045"}
Sep 11 18:30:26.245936 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:26.245390    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8"
Sep 11 18:30:26.247143 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:26.247111    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4" event={"ID":"281d75e1-25ac-4c52-bad4-14f88222061e","Type":"ContainerStarted","Data":"ee2832db80af42390d65ac01d8a85a618f6c3185aa561234cb889ca2b30159e6"}
Sep 11 18:30:26.247282 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:26.247163    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4" event={"ID":"281d75e1-25ac-4c52-bad4-14f88222061e","Type":"ContainerStarted","Data":"09074c67edf5936aa00004ef101eeda889a0198b112785f90e705fcd9922e6b0"}
Sep 11 18:30:26.247343 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:26.247305    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4"
Sep 11 18:30:26.256443 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:26.256421    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx"
Sep 11 18:30:26.267036 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:26.266982    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8" podStartSLOduration=1.2669620639999999 podStartE2EDuration="1.266962064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:26.265551994 +0000 UTC m=+1224.866913160" watchObservedRunningTime="2026-09-11 18:30:26.266962064 +0000 UTC m=+1224.868323228"
Sep 11 18:30:26.290838 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:26.290785    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4" podStartSLOduration=1.290769732 podStartE2EDuration="1.290769732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:26.28836583 +0000 UTC m=+1224.889726994" watchObservedRunningTime="2026-09-11 18:30:26.290769732 +0000 UTC m=+1224.892130895"
Sep 11 18:30:26.536208 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:26.536114    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx"]
Sep 11 18:30:26.540027 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:30:26.539986    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e4fb425_8f69_41f9_8eab_33a44f90a28a.slice/crio-95fc20a36ed8ad8be8f5d34300de66d685bdb41901e18be58ed517d021fdcb92 WatchSource:0}: Error finding container 95fc20a36ed8ad8be8f5d34300de66d685bdb41901e18be58ed517d021fdcb92: Status 404 returned error can't find the container with id 95fc20a36ed8ad8be8f5d34300de66d685bdb41901e18be58ed517d021fdcb92
Sep 11 18:30:27.255941 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:27.255895    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx" event={"ID":"1e4fb425-8f69-41f9-8eab-33a44f90a28a","Type":"ContainerStarted","Data":"e12190c1d10ebe3297b9a00f0b56d8b5dc5e41f742445b94a0d8aad26248ba00"}
Sep 11 18:30:27.256432 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:27.255961    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx" event={"ID":"1e4fb425-8f69-41f9-8eab-33a44f90a28a","Type":"ContainerStarted","Data":"95fc20a36ed8ad8be8f5d34300de66d685bdb41901e18be58ed517d021fdcb92"}
Sep 11 18:30:27.256432 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:27.256188    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx"
Sep 11 18:30:27.279108 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:27.279025    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx" podStartSLOduration=2.27900559 podStartE2EDuration="2.27900559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:27.278912618 +0000 UTC m=+1225.880273784" watchObservedRunningTime="2026-09-11 18:30:27.27900559 +0000 UTC m=+1225.880366755"
Sep 11 18:30:37.258637 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:37.258601    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4"
Sep 11 18:30:37.260836 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:37.258687    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8"
Sep 11 18:30:38.262938 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:38.262904    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx"
Sep 11 18:30:48.736995 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:48.736954    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:30:55.654575 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:55.654498    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296"]
Sep 11 18:30:55.658209 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:55.658194    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296"
Sep 11 18:30:55.660682 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:55.660656    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-5faf38\""
Sep 11 18:30:55.660812 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:55.660761    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-5faf38-dockercfg-fktvr\""
Sep 11 18:30:55.673795 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:55.673773    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4"]
Sep 11 18:30:55.677052 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:55.677032    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296"]
Sep 11 18:30:55.677173 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:55.677109    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4"
Sep 11 18:30:55.693269 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:55.693250    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4"]
Sep 11 18:30:55.822681 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:55.822647    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69daf756-4c93-467f-ad33-4b7ed7caa74a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296\" (UID: \"69daf756-4c93-467f-ad33-4b7ed7caa74a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296"
Sep 11 18:30:55.822844 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:55.822692    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4\" (UID: \"c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4"
Sep 11 18:30:55.822844 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:55.822771    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnf6s\" (UniqueName: \"kubernetes.io/projected/69daf756-4c93-467f-ad33-4b7ed7caa74a-kube-api-access-fnf6s\") pod \"payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296\" (UID: \"69daf756-4c93-467f-ad33-4b7ed7caa74a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296"
Sep 11 18:30:55.822844 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:55.822816    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6fkd\" (UniqueName: \"kubernetes.io/projected/c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5-kube-api-access-j6fkd\") pod \"payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4\" (UID: \"c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4"
Sep 11 18:30:55.923995 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:55.923910    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnf6s\" (UniqueName: \"kubernetes.io/projected/69daf756-4c93-467f-ad33-4b7ed7caa74a-kube-api-access-fnf6s\") pod \"payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296\" (UID: \"69daf756-4c93-467f-ad33-4b7ed7caa74a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296"
Sep 11 18:30:55.923995 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:55.923952    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6fkd\" (UniqueName: \"kubernetes.io/projected/c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5-kube-api-access-j6fkd\") pod \"payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4\" (UID: \"c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4"
Sep 11 18:30:55.923995 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:55.923995    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69daf756-4c93-467f-ad33-4b7ed7caa74a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296\" (UID: \"69daf756-4c93-467f-ad33-4b7ed7caa74a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296"
Sep 11 18:30:55.924218 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:55.924159    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4\" (UID: \"c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4"
Sep 11 18:30:55.924674 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:55.924658    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69daf756-4c93-467f-ad33-4b7ed7caa74a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296\" (UID: \"69daf756-4c93-467f-ad33-4b7ed7caa74a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296"
Sep 11 18:30:55.924830 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:55.924815    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4\" (UID: \"c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4"
Sep 11 18:30:55.932043 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:55.932025    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnf6s\" (UniqueName: \"kubernetes.io/projected/69daf756-4c93-467f-ad33-4b7ed7caa74a-kube-api-access-fnf6s\") pod \"payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296\" (UID: \"69daf756-4c93-467f-ad33-4b7ed7caa74a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296"
Sep 11 18:30:55.932182 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:55.932162    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6fkd\" (UniqueName: \"kubernetes.io/projected/c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5-kube-api-access-j6fkd\") pod \"payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4\" (UID: \"c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4"
Sep 11 18:30:55.968957 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:55.968927    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296"
Sep 11 18:30:55.989201 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:55.989176    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4"
Sep 11 18:30:56.248107 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:56.248082    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296"]
Sep 11 18:30:56.250253 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:30:56.250231    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69daf756_4c93_467f_ad33_4b7ed7caa74a.slice/crio-47ac10202f6e9e83c4d514f44b8e630be3376d5b5fe4b210c0d242114fc1dc88 WatchSource:0}: Error finding container 47ac10202f6e9e83c4d514f44b8e630be3376d5b5fe4b210c0d242114fc1dc88: Status 404 returned error can't find the container with id 47ac10202f6e9e83c4d514f44b8e630be3376d5b5fe4b210c0d242114fc1dc88
Sep 11 18:30:56.282379 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:56.282348    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4"]
Sep 11 18:30:56.285682 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:30:56.285659    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d4490d_4b67_45ef_90e9_d1fe1dd43ac5.slice/crio-0101a2de7edcd01dc89d227fdc30cbcc2c84ecd82b3bf0835900de160e33986c WatchSource:0}: Error finding container 0101a2de7edcd01dc89d227fdc30cbcc2c84ecd82b3bf0835900de160e33986c: Status 404 returned error can't find the container with id 0101a2de7edcd01dc89d227fdc30cbcc2c84ecd82b3bf0835900de160e33986c
Sep 11 18:30:56.404943 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:56.404904    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4" event={"ID":"c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5","Type":"ContainerStarted","Data":"f9b63828894d93ab8f20d85c42d2d399b9f28a52c04bf6049d617acb88ae9b2a"}
Sep 11 18:30:56.405134 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:56.404967    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4" event={"ID":"c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5","Type":"ContainerStarted","Data":"0101a2de7edcd01dc89d227fdc30cbcc2c84ecd82b3bf0835900de160e33986c"}
Sep 11 18:30:56.405134 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:56.405084    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4"
Sep 11 18:30:56.406542 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:56.406518    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296" event={"ID":"69daf756-4c93-467f-ad33-4b7ed7caa74a","Type":"ContainerStarted","Data":"c6b359c8942b669aa1f6ff3258b6e1e68185ecaef3a7cd3c4adf570c431194e1"}
Sep 11 18:30:56.406655 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:56.406557    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296" event={"ID":"69daf756-4c93-467f-ad33-4b7ed7caa74a","Type":"ContainerStarted","Data":"47ac10202f6e9e83c4d514f44b8e630be3376d5b5fe4b210c0d242114fc1dc88"}
Sep 11 18:30:56.406655 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:56.406576    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296"
Sep 11 18:30:56.425956 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:56.425915    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4" podStartSLOduration=1.425901408 podStartE2EDuration="1.425901408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:56.424339044 +0000 UTC m=+1255.025700207" watchObservedRunningTime="2026-09-11 18:30:56.425901408 +0000 UTC m=+1255.027262570"
Sep 11 18:30:56.448245 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:30:56.448181    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296" podStartSLOduration=1.44816154 podStartE2EDuration="1.44816154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:56.447505574 +0000 UTC m=+1255.048866738" watchObservedRunningTime="2026-09-11 18:30:56.44816154 +0000 UTC m=+1255.049522703"
Sep 11 18:31:07.414128 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:07.414094    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296"
Sep 11 18:31:07.414521 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:07.414210    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4"
Sep 11 18:31:09.109598 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:09.109549    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:31:31.504595 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:31.504559    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:31:34.495316 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.495284    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"]
Sep 11 18:31:34.498702 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.498684    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:31:34.502012 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.501986    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3e24a782\"/\"openshift-service-ca.crt\""
Sep 11 18:31:34.502158 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.502020    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3e24a782\"/\"e2e-lifecycle-model-3e24a782-kserve-self-signed-certs\""
Sep 11 18:31:34.502158 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.501986    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3e24a782\"/\"default-dockercfg-bc9d5\""
Sep 11 18:31:34.502158 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.502085    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3e24a782\"/\"kube-root-ca.crt\""
Sep 11 18:31:34.509784 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.509763    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"]
Sep 11 18:31:34.566501 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.566462    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9ae30e6-f378-45b5-b382-d5e7568c46e0-model-cache\") pod \"e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5\" (UID: \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\") " pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:31:34.566501 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.566497    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6txxl\" (UniqueName: \"kubernetes.io/projected/f9ae30e6-f378-45b5-b382-d5e7568c46e0-kube-api-access-6txxl\") pod \"e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5\" (UID: \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\") " pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:31:34.566735 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.566526    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9ae30e6-f378-45b5-b382-d5e7568c46e0-tmp-dir\") pod \"e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5\" (UID: \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\") " pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:31:34.566735 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.566545    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9ae30e6-f378-45b5-b382-d5e7568c46e0-tls-certs\") pod \"e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5\" (UID: \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\") " pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:31:34.566735 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.566655    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9ae30e6-f378-45b5-b382-d5e7568c46e0-dshm\") pod \"e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5\" (UID: \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\") " pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:31:34.566735 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.566708    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9ae30e6-f378-45b5-b382-d5e7568c46e0-home\") pod \"e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5\" (UID: \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\") " pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:31:34.667364 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.667331    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9ae30e6-f378-45b5-b382-d5e7568c46e0-model-cache\") pod \"e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5\" (UID: \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\") " pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:31:34.667364 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.667367    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6txxl\" (UniqueName: \"kubernetes.io/projected/f9ae30e6-f378-45b5-b382-d5e7568c46e0-kube-api-access-6txxl\") pod \"e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5\" (UID: \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\") " pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:31:34.667619 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.667396    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9ae30e6-f378-45b5-b382-d5e7568c46e0-tmp-dir\") pod \"e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5\" (UID: \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\") " pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:31:34.667619 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.667413    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9ae30e6-f378-45b5-b382-d5e7568c46e0-tls-certs\") pod \"e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5\" (UID: \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\") " pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:31:34.667619 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.667451    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9ae30e6-f378-45b5-b382-d5e7568c46e0-dshm\") pod \"e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5\" (UID: \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\") " pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:31:34.667619 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.667484    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9ae30e6-f378-45b5-b382-d5e7568c46e0-home\") pod \"e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5\" (UID: \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\") " pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:31:34.667903 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.667877    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9ae30e6-f378-45b5-b382-d5e7568c46e0-tmp-dir\") pod \"e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5\" (UID: \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\") " pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:31:34.667961 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.667938    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9ae30e6-f378-45b5-b382-d5e7568c46e0-home\") pod \"e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5\" (UID: \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\") " pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:31:34.668002 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.667966    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9ae30e6-f378-45b5-b382-d5e7568c46e0-model-cache\") pod \"e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5\" (UID: \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\") " pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:31:34.669801 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.669777    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9ae30e6-f378-45b5-b382-d5e7568c46e0-dshm\") pod \"e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5\" (UID: \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\") " pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:31:34.669957 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.669942    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9ae30e6-f378-45b5-b382-d5e7568c46e0-tls-certs\") pod \"e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5\" (UID: \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\") " pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:31:34.675775 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.675751    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6txxl\" (UniqueName: \"kubernetes.io/projected/f9ae30e6-f378-45b5-b382-d5e7568c46e0-kube-api-access-6txxl\") pod \"e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5\" (UID: \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\") " pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:31:34.809579 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:34.809493    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:31:35.090521 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:35.090496    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"]
Sep 11 18:31:35.092388 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:31:35.092358    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9ae30e6_f378_45b5_b382_d5e7568c46e0.slice/crio-55d3fbc46537f2691992d3695dd75035ffaeedaaf48a237d88191e1d07d558a5 WatchSource:0}: Error finding container 55d3fbc46537f2691992d3695dd75035ffaeedaaf48a237d88191e1d07d558a5: Status 404 returned error can't find the container with id 55d3fbc46537f2691992d3695dd75035ffaeedaaf48a237d88191e1d07d558a5
Sep 11 18:31:35.607339 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:35.607291    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5" event={"ID":"f9ae30e6-f378-45b5-b382-d5e7568c46e0","Type":"ContainerStarted","Data":"6b75a1ce01b4a7ea0719db37b575643779e1e7083e979ef3ddc025a27a5fad3f"}
Sep 11 18:31:35.607777 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:35.607348    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5" event={"ID":"f9ae30e6-f378-45b5-b382-d5e7568c46e0","Type":"ContainerStarted","Data":"55d3fbc46537f2691992d3695dd75035ffaeedaaf48a237d88191e1d07d558a5"}
Sep 11 18:31:35.629743 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:35.629686    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5" podStartSLOduration=1.6296684799999999 podStartE2EDuration="1.62966848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:31:35.628170076 +0000 UTC m=+1294.229531222" watchObservedRunningTime="2026-09-11 18:31:35.62966848 +0000 UTC m=+1294.231029645"
Sep 11 18:31:44.810082 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:44.810029    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:31:44.810548 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:44.810105    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:31:44.822756 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:44.822728    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:31:45.674704 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:45.674671    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:31:54.512802 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:31:54.512760    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:32:14.400091 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.400060    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"]
Sep 11 18:32:14.403637 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.403606    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:32:14.406617 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.406592    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bf9898\"/\"default-dockercfg-b87pn\""
Sep 11 18:32:14.406755 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.406646    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bf9898\"/\"kube-root-ca.crt\""
Sep 11 18:32:14.406755 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.406649    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bf9898\"/\"auth-test-model-bf9898-kserve-self-signed-certs\""
Sep 11 18:32:14.406755 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.406619    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bf9898\"/\"openshift-service-ca.crt\""
Sep 11 18:32:14.414366 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.414342    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"]
Sep 11 18:32:14.534945 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.534901    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b871f0-6d44-437c-9d58-d58f80a85b4f-model-cache\") pod \"auth-test-model-bf9898-kserve-65877c4dcf-wdttc\" (UID: \"44b871f0-6d44-437c-9d58-d58f80a85b4f\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:32:14.535120 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.535012    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b871f0-6d44-437c-9d58-d58f80a85b4f-dshm\") pod \"auth-test-model-bf9898-kserve-65877c4dcf-wdttc\" (UID: \"44b871f0-6d44-437c-9d58-d58f80a85b4f\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:32:14.535120 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.535043    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b871f0-6d44-437c-9d58-d58f80a85b4f-tls-certs\") pod \"auth-test-model-bf9898-kserve-65877c4dcf-wdttc\" (UID: \"44b871f0-6d44-437c-9d58-d58f80a85b4f\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:32:14.535209 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.535128    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86nzb\" (UniqueName: \"kubernetes.io/projected/44b871f0-6d44-437c-9d58-d58f80a85b4f-kube-api-access-86nzb\") pod \"auth-test-model-bf9898-kserve-65877c4dcf-wdttc\" (UID: \"44b871f0-6d44-437c-9d58-d58f80a85b4f\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:32:14.535209 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.535157    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b871f0-6d44-437c-9d58-d58f80a85b4f-home\") pod \"auth-test-model-bf9898-kserve-65877c4dcf-wdttc\" (UID: \"44b871f0-6d44-437c-9d58-d58f80a85b4f\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:32:14.535209 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.535176    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b871f0-6d44-437c-9d58-d58f80a85b4f-tmp-dir\") pod \"auth-test-model-bf9898-kserve-65877c4dcf-wdttc\" (UID: \"44b871f0-6d44-437c-9d58-d58f80a85b4f\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:32:14.636235 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.636199    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b871f0-6d44-437c-9d58-d58f80a85b4f-dshm\") pod \"auth-test-model-bf9898-kserve-65877c4dcf-wdttc\" (UID: \"44b871f0-6d44-437c-9d58-d58f80a85b4f\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:32:14.636235 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.636238    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b871f0-6d44-437c-9d58-d58f80a85b4f-tls-certs\") pod \"auth-test-model-bf9898-kserve-65877c4dcf-wdttc\" (UID: \"44b871f0-6d44-437c-9d58-d58f80a85b4f\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:32:14.636481 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.636289    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86nzb\" (UniqueName: \"kubernetes.io/projected/44b871f0-6d44-437c-9d58-d58f80a85b4f-kube-api-access-86nzb\") pod \"auth-test-model-bf9898-kserve-65877c4dcf-wdttc\" (UID: \"44b871f0-6d44-437c-9d58-d58f80a85b4f\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:32:14.636481 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.636372    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b871f0-6d44-437c-9d58-d58f80a85b4f-home\") pod \"auth-test-model-bf9898-kserve-65877c4dcf-wdttc\" (UID: \"44b871f0-6d44-437c-9d58-d58f80a85b4f\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:32:14.636481 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.636400    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b871f0-6d44-437c-9d58-d58f80a85b4f-tmp-dir\") pod \"auth-test-model-bf9898-kserve-65877c4dcf-wdttc\" (UID: \"44b871f0-6d44-437c-9d58-d58f80a85b4f\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:32:14.636481 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.636441    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b871f0-6d44-437c-9d58-d58f80a85b4f-model-cache\") pod \"auth-test-model-bf9898-kserve-65877c4dcf-wdttc\" (UID: \"44b871f0-6d44-437c-9d58-d58f80a85b4f\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:32:14.636789 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.636763    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b871f0-6d44-437c-9d58-d58f80a85b4f-home\") pod \"auth-test-model-bf9898-kserve-65877c4dcf-wdttc\" (UID: \"44b871f0-6d44-437c-9d58-d58f80a85b4f\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:32:14.636929 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.636825    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b871f0-6d44-437c-9d58-d58f80a85b4f-tmp-dir\") pod \"auth-test-model-bf9898-kserve-65877c4dcf-wdttc\" (UID: \"44b871f0-6d44-437c-9d58-d58f80a85b4f\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:32:14.636929 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.636881    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b871f0-6d44-437c-9d58-d58f80a85b4f-model-cache\") pod \"auth-test-model-bf9898-kserve-65877c4dcf-wdttc\" (UID: \"44b871f0-6d44-437c-9d58-d58f80a85b4f\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:32:14.638651 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.638631    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b871f0-6d44-437c-9d58-d58f80a85b4f-dshm\") pod \"auth-test-model-bf9898-kserve-65877c4dcf-wdttc\" (UID: \"44b871f0-6d44-437c-9d58-d58f80a85b4f\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:32:14.638915 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.638895    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b871f0-6d44-437c-9d58-d58f80a85b4f-tls-certs\") pod \"auth-test-model-bf9898-kserve-65877c4dcf-wdttc\" (UID: \"44b871f0-6d44-437c-9d58-d58f80a85b4f\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:32:14.643987 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.643965    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86nzb\" (UniqueName: \"kubernetes.io/projected/44b871f0-6d44-437c-9d58-d58f80a85b4f-kube-api-access-86nzb\") pod \"auth-test-model-bf9898-kserve-65877c4dcf-wdttc\" (UID: \"44b871f0-6d44-437c-9d58-d58f80a85b4f\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:32:14.715527 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.715498    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:32:14.987912 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:14.987887    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"]
Sep 11 18:32:14.990259 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:32:14.990233    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44b871f0_6d44_437c_9d58_d58f80a85b4f.slice/crio-f5387f2bb6d569bf86abac1ff5d08af0a60121b6723491616e39ff5fadf1076c WatchSource:0}: Error finding container f5387f2bb6d569bf86abac1ff5d08af0a60121b6723491616e39ff5fadf1076c: Status 404 returned error can't find the container with id f5387f2bb6d569bf86abac1ff5d08af0a60121b6723491616e39ff5fadf1076c
Sep 11 18:32:15.815317 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:15.815284    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc" event={"ID":"44b871f0-6d44-437c-9d58-d58f80a85b4f","Type":"ContainerStarted","Data":"c5b8672cf5ed7e30a94fbdb337c62cfa7db7878809cc58dcc734c998e22373bf"}
Sep 11 18:32:15.815317 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:15.815325    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc" event={"ID":"44b871f0-6d44-437c-9d58-d58f80a85b4f","Type":"ContainerStarted","Data":"f5387f2bb6d569bf86abac1ff5d08af0a60121b6723491616e39ff5fadf1076c"}
Sep 11 18:32:15.832964 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:15.832911    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc" podStartSLOduration=1.832896026 podStartE2EDuration="1.832896026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:32:15.831447223 +0000 UTC m=+1334.432808388" watchObservedRunningTime="2026-09-11 18:32:15.832896026 +0000 UTC m=+1334.434257189"
Sep 11 18:32:18.006638 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:18.006602    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"]
Sep 11 18:32:24.716264 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:24.716178    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:32:24.716264 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:24.716232    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:32:24.728955 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:24.728921    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:32:24.878902 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:24.878864    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:32:28.714079 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:28.714035    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:32:33.026284 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.026232    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd" podUID="e373bbfd-9aab-4552-83d1-6dfdd4685563" containerName="main" containerID="cri-o://e54b704511db34af82f281d97d9262a19430ad4ae554a8268adcaad191e12367" gracePeriod=45
Sep 11 18:32:33.274483 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.274457    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:32:33.409395 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.409307    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e373bbfd-9aab-4552-83d1-6dfdd4685563-tmp-dir\") pod \"e373bbfd-9aab-4552-83d1-6dfdd4685563\" (UID: \"e373bbfd-9aab-4552-83d1-6dfdd4685563\") "
Sep 11 18:32:33.409395 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.409342    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e373bbfd-9aab-4552-83d1-6dfdd4685563-dshm\") pod \"e373bbfd-9aab-4552-83d1-6dfdd4685563\" (UID: \"e373bbfd-9aab-4552-83d1-6dfdd4685563\") "
Sep 11 18:32:33.409395 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.409379    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zt7mj\" (UniqueName: \"kubernetes.io/projected/e373bbfd-9aab-4552-83d1-6dfdd4685563-kube-api-access-zt7mj\") pod \"e373bbfd-9aab-4552-83d1-6dfdd4685563\" (UID: \"e373bbfd-9aab-4552-83d1-6dfdd4685563\") "
Sep 11 18:32:33.409395 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.409400    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e373bbfd-9aab-4552-83d1-6dfdd4685563-tls-certs\") pod \"e373bbfd-9aab-4552-83d1-6dfdd4685563\" (UID: \"e373bbfd-9aab-4552-83d1-6dfdd4685563\") "
Sep 11 18:32:33.409777 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.409443    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e373bbfd-9aab-4552-83d1-6dfdd4685563-model-cache\") pod \"e373bbfd-9aab-4552-83d1-6dfdd4685563\" (UID: \"e373bbfd-9aab-4552-83d1-6dfdd4685563\") "
Sep 11 18:32:33.409777 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.409521    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e373bbfd-9aab-4552-83d1-6dfdd4685563-home\") pod \"e373bbfd-9aab-4552-83d1-6dfdd4685563\" (UID: \"e373bbfd-9aab-4552-83d1-6dfdd4685563\") "
Sep 11 18:32:33.409777 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.409697    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e373bbfd-9aab-4552-83d1-6dfdd4685563-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e373bbfd-9aab-4552-83d1-6dfdd4685563" (UID: "e373bbfd-9aab-4552-83d1-6dfdd4685563"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:32:33.409934 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.409811    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e373bbfd-9aab-4552-83d1-6dfdd4685563-model-cache" (OuterVolumeSpecName: "model-cache") pod "e373bbfd-9aab-4552-83d1-6dfdd4685563" (UID: "e373bbfd-9aab-4552-83d1-6dfdd4685563"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:32:33.409934 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.409860    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e373bbfd-9aab-4552-83d1-6dfdd4685563-home" (OuterVolumeSpecName: "home") pod "e373bbfd-9aab-4552-83d1-6dfdd4685563" (UID: "e373bbfd-9aab-4552-83d1-6dfdd4685563"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:32:33.411761 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.411736    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e373bbfd-9aab-4552-83d1-6dfdd4685563-dshm" (OuterVolumeSpecName: "dshm") pod "e373bbfd-9aab-4552-83d1-6dfdd4685563" (UID: "e373bbfd-9aab-4552-83d1-6dfdd4685563"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:32:33.411875 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.411796    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e373bbfd-9aab-4552-83d1-6dfdd4685563-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e373bbfd-9aab-4552-83d1-6dfdd4685563" (UID: "e373bbfd-9aab-4552-83d1-6dfdd4685563"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:32:33.411875 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.411799    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e373bbfd-9aab-4552-83d1-6dfdd4685563-kube-api-access-zt7mj" (OuterVolumeSpecName: "kube-api-access-zt7mj") pod "e373bbfd-9aab-4552-83d1-6dfdd4685563" (UID: "e373bbfd-9aab-4552-83d1-6dfdd4685563"). InnerVolumeSpecName "kube-api-access-zt7mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:32:33.510546 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.510510    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e373bbfd-9aab-4552-83d1-6dfdd4685563-tls-certs\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:32:33.510546 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.510542    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e373bbfd-9aab-4552-83d1-6dfdd4685563-model-cache\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:32:33.510546 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.510552    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e373bbfd-9aab-4552-83d1-6dfdd4685563-home\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:32:33.510812 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.510561    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e373bbfd-9aab-4552-83d1-6dfdd4685563-tmp-dir\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:32:33.510812 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.510568    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e373bbfd-9aab-4552-83d1-6dfdd4685563-dshm\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:32:33.510812 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.510576    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt7mj\" (UniqueName: \"kubernetes.io/projected/e373bbfd-9aab-4552-83d1-6dfdd4685563-kube-api-access-zt7mj\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:32:33.912064 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.912025    2576 generic.go:356] "Generic (PLEG): container finished" podID="e373bbfd-9aab-4552-83d1-6dfdd4685563" containerID="e54b704511db34af82f281d97d9262a19430ad4ae554a8268adcaad191e12367" exitCode=0
Sep 11 18:32:33.912266 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.912066    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"
Sep 11 18:32:33.912266 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.912099    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd" event={"ID":"e373bbfd-9aab-4552-83d1-6dfdd4685563","Type":"ContainerDied","Data":"e54b704511db34af82f281d97d9262a19430ad4ae554a8268adcaad191e12367"}
Sep 11 18:32:33.912266 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.912138    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd" event={"ID":"e373bbfd-9aab-4552-83d1-6dfdd4685563","Type":"ContainerDied","Data":"1315a055751b4118fbe359564aa7cb47dace9f9758f585320607c3ae57791ed6"}
Sep 11 18:32:33.912266 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.912156    2576 scope.go:117] "RemoveContainer" containerID="e54b704511db34af82f281d97d9262a19430ad4ae554a8268adcaad191e12367"
Sep 11 18:32:33.922111 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.922093    2576 scope.go:117] "RemoveContainer" containerID="e54b704511db34af82f281d97d9262a19430ad4ae554a8268adcaad191e12367"
Sep 11 18:32:33.922379 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:32:33.922362    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e54b704511db34af82f281d97d9262a19430ad4ae554a8268adcaad191e12367\": container with ID starting with e54b704511db34af82f281d97d9262a19430ad4ae554a8268adcaad191e12367 not found: ID does not exist" containerID="e54b704511db34af82f281d97d9262a19430ad4ae554a8268adcaad191e12367"
Sep 11 18:32:33.922437 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.922387    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e54b704511db34af82f281d97d9262a19430ad4ae554a8268adcaad191e12367"} err="failed to get container status \"e54b704511db34af82f281d97d9262a19430ad4ae554a8268adcaad191e12367\": rpc error: code = NotFound desc = could not find container \"e54b704511db34af82f281d97d9262a19430ad4ae554a8268adcaad191e12367\": container with ID starting with e54b704511db34af82f281d97d9262a19430ad4ae554a8268adcaad191e12367 not found: ID does not exist"
Sep 11 18:32:33.935648 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.935622    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"]
Sep 11 18:32:33.938930 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:33.938909    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-553596/e2e-conflict-b-c17caa91-kserve-6b46797b65-7jwgd"]
Sep 11 18:32:35.910943 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:35.910904    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e373bbfd-9aab-4552-83d1-6dfdd4685563" path="/var/lib/kubelet/pods/e373bbfd-9aab-4552-83d1-6dfdd4685563/volumes"
Sep 11 18:32:51.930725 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:51.930692    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e24a782-wmvrf"]
Sep 11 18:32:51.931188 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:51.931072    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e373bbfd-9aab-4552-83d1-6dfdd4685563" containerName="main"
Sep 11 18:32:51.931188 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:51.931084    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e373bbfd-9aab-4552-83d1-6dfdd4685563" containerName="main"
Sep 11 18:32:51.931188 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:51.931146    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e373bbfd-9aab-4552-83d1-6dfdd4685563" containerName="main"
Sep 11 18:32:51.935469 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:51.935449    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e24a782-wmvrf"
Sep 11 18:32:51.937758 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:51.937737    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t9kdx\""
Sep 11 18:32:51.941661 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:51.941483    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e24a782-wmvrf"]
Sep 11 18:32:51.979505 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:51.979469    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7091bb6b-be16-4752-91c2-4c584205a190-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3e24a782-wmvrf\" (UID: \"7091bb6b-be16-4752-91c2-4c584205a190\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e24a782-wmvrf"
Sep 11 18:32:52.080501 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:52.080466    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7091bb6b-be16-4752-91c2-4c584205a190-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3e24a782-wmvrf\" (UID: \"7091bb6b-be16-4752-91c2-4c584205a190\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e24a782-wmvrf"
Sep 11 18:32:52.081092 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:52.081072    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7091bb6b-be16-4752-91c2-4c584205a190-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3e24a782-wmvrf\" (UID: \"7091bb6b-be16-4752-91c2-4c584205a190\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e24a782-wmvrf"
Sep 11 18:32:52.248405 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:52.248376    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e24a782-wmvrf"
Sep 11 18:32:52.518681 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:52.518652    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e24a782-wmvrf"]
Sep 11 18:32:52.520831 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:32:52.520796    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7091bb6b_be16_4752_91c2_4c584205a190.slice/crio-0214253d7e4612c06ef761422a8c3c354cc807ace88a3f6a05679b9a7cd56401 WatchSource:0}: Error finding container 0214253d7e4612c06ef761422a8c3c354cc807ace88a3f6a05679b9a7cd56401: Status 404 returned error can't find the container with id 0214253d7e4612c06ef761422a8c3c354cc807ace88a3f6a05679b9a7cd56401
Sep 11 18:32:53.005676 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:53.005640    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e24a782-wmvrf" event={"ID":"7091bb6b-be16-4752-91c2-4c584205a190","Type":"ContainerStarted","Data":"d6bcc2b80de445fc0941750ec9c7d4f09f102d4aa4d1ef6432b2c7ac4fa606f5"}
Sep 11 18:32:53.006051 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:53.005692    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e24a782-wmvrf" event={"ID":"7091bb6b-be16-4752-91c2-4c584205a190","Type":"ContainerStarted","Data":"0214253d7e4612c06ef761422a8c3c354cc807ace88a3f6a05679b9a7cd56401"}
Sep 11 18:32:53.022983 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:53.022933    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e24a782-wmvrf" podStartSLOduration=2.02291794 podStartE2EDuration="2.02291794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:32:53.020535527 +0000 UTC m=+1371.621896692" watchObservedRunningTime="2026-09-11 18:32:53.02291794 +0000 UTC m=+1371.624279104"
Sep 11 18:32:54.011912 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:54.011880    2576 generic.go:356] "Generic (PLEG): container finished" podID="7091bb6b-be16-4752-91c2-4c584205a190" containerID="d6bcc2b80de445fc0941750ec9c7d4f09f102d4aa4d1ef6432b2c7ac4fa606f5" exitCode=0
Sep 11 18:32:54.012293 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:54.011966    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e24a782-wmvrf" event={"ID":"7091bb6b-be16-4752-91c2-4c584205a190","Type":"ContainerDied","Data":"d6bcc2b80de445fc0941750ec9c7d4f09f102d4aa4d1ef6432b2c7ac4fa606f5"}
Sep 11 18:32:55.150752 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:55.150726    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e24a782-wmvrf"
Sep 11 18:32:55.210212 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:55.210180    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7091bb6b-be16-4752-91c2-4c584205a190-maas-api-service-ca\") pod \"7091bb6b-be16-4752-91c2-4c584205a190\" (UID: \"7091bb6b-be16-4752-91c2-4c584205a190\") "
Sep 11 18:32:55.210528 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:55.210502    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7091bb6b-be16-4752-91c2-4c584205a190-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7091bb6b-be16-4752-91c2-4c584205a190" (UID: "7091bb6b-be16-4752-91c2-4c584205a190"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:32:55.311111 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:55.311030    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7091bb6b-be16-4752-91c2-4c584205a190-maas-api-service-ca\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:32:56.024123 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:56.024090    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e24a782-wmvrf"
Sep 11 18:32:56.024307 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:56.024124    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e24a782-wmvrf" event={"ID":"7091bb6b-be16-4752-91c2-4c584205a190","Type":"ContainerDied","Data":"0214253d7e4612c06ef761422a8c3c354cc807ace88a3f6a05679b9a7cd56401"}
Sep 11 18:32:56.024307 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:56.024187    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0214253d7e4612c06ef761422a8c3c354cc807ace88a3f6a05679b9a7cd56401"
Sep 11 18:32:56.992493 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:56.992462    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"]
Sep 11 18:32:56.993013 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:56.992986    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7091bb6b-be16-4752-91c2-4c584205a190" containerName="revoke-keys"
Sep 11 18:32:56.993013 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:56.993012    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7091bb6b-be16-4752-91c2-4c584205a190" containerName="revoke-keys"
Sep 11 18:32:56.993205 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:56.993132    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7091bb6b-be16-4752-91c2-4c584205a190" containerName="revoke-keys"
Sep 11 18:32:56.996398 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:56.996380    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:32:56.999776 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:56.999755    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5faf38\"/\"default-dockercfg-qwk46\""
Sep 11 18:32:56.999894 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:56.999820    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5faf38\"/\"kube-root-ca.crt\""
Sep 11 18:32:56.999894 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:56.999861    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5faf38\"/\"openshift-service-ca.crt\""
Sep 11 18:32:57.000013 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:56.999754    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5faf38\"/\"auth-test-model-5faf38-kserve-self-signed-certs\""
Sep 11 18:32:57.005564 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:57.005542    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"]
Sep 11 18:32:57.126714 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:57.126683    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47476a97-dd25-4605-8558-cdc821679b74-home\") pod \"auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn\" (UID: \"47476a97-dd25-4605-8558-cdc821679b74\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:32:57.126858 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:57.126733    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47476a97-dd25-4605-8558-cdc821679b74-tmp-dir\") pod \"auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn\" (UID: \"47476a97-dd25-4605-8558-cdc821679b74\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:32:57.126858 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:57.126771    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47476a97-dd25-4605-8558-cdc821679b74-dshm\") pod \"auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn\" (UID: \"47476a97-dd25-4605-8558-cdc821679b74\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:32:57.126858 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:57.126792    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47476a97-dd25-4605-8558-cdc821679b74-model-cache\") pod \"auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn\" (UID: \"47476a97-dd25-4605-8558-cdc821679b74\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:32:57.126858 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:57.126840    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47476a97-dd25-4605-8558-cdc821679b74-tls-certs\") pod \"auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn\" (UID: \"47476a97-dd25-4605-8558-cdc821679b74\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:32:57.127009 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:57.126896    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wm4k\" (UniqueName: \"kubernetes.io/projected/47476a97-dd25-4605-8558-cdc821679b74-kube-api-access-9wm4k\") pod \"auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn\" (UID: \"47476a97-dd25-4605-8558-cdc821679b74\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:32:57.227333 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:57.227300    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47476a97-dd25-4605-8558-cdc821679b74-home\") pod \"auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn\" (UID: \"47476a97-dd25-4605-8558-cdc821679b74\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:32:57.227539 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:57.227349    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47476a97-dd25-4605-8558-cdc821679b74-tmp-dir\") pod \"auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn\" (UID: \"47476a97-dd25-4605-8558-cdc821679b74\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:32:57.227539 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:57.227367    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47476a97-dd25-4605-8558-cdc821679b74-dshm\") pod \"auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn\" (UID: \"47476a97-dd25-4605-8558-cdc821679b74\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:32:57.227539 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:57.227482    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47476a97-dd25-4605-8558-cdc821679b74-model-cache\") pod \"auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn\" (UID: \"47476a97-dd25-4605-8558-cdc821679b74\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:32:57.227539 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:57.227536    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47476a97-dd25-4605-8558-cdc821679b74-tls-certs\") pod \"auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn\" (UID: \"47476a97-dd25-4605-8558-cdc821679b74\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:32:57.227787 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:57.227617    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wm4k\" (UniqueName: \"kubernetes.io/projected/47476a97-dd25-4605-8558-cdc821679b74-kube-api-access-9wm4k\") pod \"auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn\" (UID: \"47476a97-dd25-4605-8558-cdc821679b74\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:32:57.227787 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:57.227717    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47476a97-dd25-4605-8558-cdc821679b74-home\") pod \"auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn\" (UID: \"47476a97-dd25-4605-8558-cdc821679b74\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:32:57.227903 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:57.227823    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47476a97-dd25-4605-8558-cdc821679b74-tmp-dir\") pod \"auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn\" (UID: \"47476a97-dd25-4605-8558-cdc821679b74\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:32:57.227959 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:57.227915    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47476a97-dd25-4605-8558-cdc821679b74-model-cache\") pod \"auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn\" (UID: \"47476a97-dd25-4605-8558-cdc821679b74\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:32:57.229634 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:57.229615    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47476a97-dd25-4605-8558-cdc821679b74-dshm\") pod \"auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn\" (UID: \"47476a97-dd25-4605-8558-cdc821679b74\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:32:57.229901 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:57.229885    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47476a97-dd25-4605-8558-cdc821679b74-tls-certs\") pod \"auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn\" (UID: \"47476a97-dd25-4605-8558-cdc821679b74\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:32:57.235461 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:57.235436    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wm4k\" (UniqueName: \"kubernetes.io/projected/47476a97-dd25-4605-8558-cdc821679b74-kube-api-access-9wm4k\") pod \"auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn\" (UID: \"47476a97-dd25-4605-8558-cdc821679b74\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:32:57.307872 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:57.307802    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:32:57.581038 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:57.581015    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"]
Sep 11 18:32:57.583155 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:32:57.583120    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47476a97_dd25_4605_8558_cdc821679b74.slice/crio-4bc51ad0acbd74246dc8197133c99f5f26d6a57ecf092315c8c10e9e399cbaef WatchSource:0}: Error finding container 4bc51ad0acbd74246dc8197133c99f5f26d6a57ecf092315c8c10e9e399cbaef: Status 404 returned error can't find the container with id 4bc51ad0acbd74246dc8197133c99f5f26d6a57ecf092315c8c10e9e399cbaef
Sep 11 18:32:58.036275 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:58.036224    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn" event={"ID":"47476a97-dd25-4605-8558-cdc821679b74","Type":"ContainerStarted","Data":"d264b2c77e5a6a01cf5f82a273288fb03a225d0e3f4920840c6dbb82a13a5690"}
Sep 11 18:32:58.036275 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:58.036279    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn" event={"ID":"47476a97-dd25-4605-8558-cdc821679b74","Type":"ContainerStarted","Data":"4bc51ad0acbd74246dc8197133c99f5f26d6a57ecf092315c8c10e9e399cbaef"}
Sep 11 18:32:58.054519 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:32:58.054472    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn" podStartSLOduration=2.054456481 podStartE2EDuration="2.054456481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:32:58.053484829 +0000 UTC m=+1376.654845992" watchObservedRunningTime="2026-09-11 18:32:58.054456481 +0000 UTC m=+1376.655817645"
Sep 11 18:33:01.010326 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:01.010291    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:33:07.308798 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:07.308750    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:33:07.309262 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:07.308814    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:33:07.321795 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:07.321754    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:33:07.548261 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:07.548218    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8"]
Sep 11 18:33:07.548558 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:07.548505    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8" podUID="078cd6e6-514e-458d-840e-415c21f14bc8" containerName="payload-processing" containerID="cri-o://fdb01ca240939349e432b9f254e2f977bf41ca7a997dabd2743059ad6b7200d8" gracePeriod=30
Sep 11 18:33:07.554844 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:07.554810    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4"]
Sep 11 18:33:07.555104 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:07.555073    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4" podUID="281d75e1-25ac-4c52-bad4-14f88222061e" containerName="payload-pre-processing" containerID="cri-o://ee2832db80af42390d65ac01d8a85a618f6c3185aa561234cb889ca2b30159e6" gracePeriod=30
Sep 11 18:33:07.735532 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:07.735495    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e24a782-wmvrf"]
Sep 11 18:33:07.738209 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:07.738095    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3e24a782-wmvrf"]
Sep 11 18:33:07.814069 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:07.814039    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8"
Sep 11 18:33:07.845899 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:07.845866    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4"
Sep 11 18:33:07.911249 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:07.911210    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7091bb6b-be16-4752-91c2-4c584205a190" path="/var/lib/kubelet/pods/7091bb6b-be16-4752-91c2-4c584205a190/volumes"
Sep 11 18:33:07.927720 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:07.927678    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrltl\" (UniqueName: \"kubernetes.io/projected/281d75e1-25ac-4c52-bad4-14f88222061e-kube-api-access-jrltl\") pod \"281d75e1-25ac-4c52-bad4-14f88222061e\" (UID: \"281d75e1-25ac-4c52-bad4-14f88222061e\") "
Sep 11 18:33:07.927720 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:07.927722    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078cd6e6-514e-458d-840e-415c21f14bc8-plugins-config-volume\") pod \"078cd6e6-514e-458d-840e-415c21f14bc8\" (UID: \"078cd6e6-514e-458d-840e-415c21f14bc8\") "
Sep 11 18:33:07.927922 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:07.927816    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281d75e1-25ac-4c52-bad4-14f88222061e-plugins-config-volume\") pod \"281d75e1-25ac-4c52-bad4-14f88222061e\" (UID: \"281d75e1-25ac-4c52-bad4-14f88222061e\") "
Sep 11 18:33:07.927922 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:07.927900    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzpxl\" (UniqueName: \"kubernetes.io/projected/078cd6e6-514e-458d-840e-415c21f14bc8-kube-api-access-wzpxl\") pod \"078cd6e6-514e-458d-840e-415c21f14bc8\" (UID: \"078cd6e6-514e-458d-840e-415c21f14bc8\") "
Sep 11 18:33:07.928202 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:07.928159    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/078cd6e6-514e-458d-840e-415c21f14bc8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "078cd6e6-514e-458d-840e-415c21f14bc8" (UID: "078cd6e6-514e-458d-840e-415c21f14bc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:33:07.928338 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:07.928176    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/281d75e1-25ac-4c52-bad4-14f88222061e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "281d75e1-25ac-4c52-bad4-14f88222061e" (UID: "281d75e1-25ac-4c52-bad4-14f88222061e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:33:07.929971 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:07.929950    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/281d75e1-25ac-4c52-bad4-14f88222061e-kube-api-access-jrltl" (OuterVolumeSpecName: "kube-api-access-jrltl") pod "281d75e1-25ac-4c52-bad4-14f88222061e" (UID: "281d75e1-25ac-4c52-bad4-14f88222061e"). InnerVolumeSpecName "kube-api-access-jrltl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:33:07.929971 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:07.929963    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078cd6e6-514e-458d-840e-415c21f14bc8-kube-api-access-wzpxl" (OuterVolumeSpecName: "kube-api-access-wzpxl") pod "078cd6e6-514e-458d-840e-415c21f14bc8" (UID: "078cd6e6-514e-458d-840e-415c21f14bc8"). InnerVolumeSpecName "kube-api-access-wzpxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:33:08.029022 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:08.028986    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrltl\" (UniqueName: \"kubernetes.io/projected/281d75e1-25ac-4c52-bad4-14f88222061e-kube-api-access-jrltl\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:33:08.029022 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:08.029017    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078cd6e6-514e-458d-840e-415c21f14bc8-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:33:08.029022 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:08.029027    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281d75e1-25ac-4c52-bad4-14f88222061e-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:33:08.029335 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:08.029040    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzpxl\" (UniqueName: \"kubernetes.io/projected/078cd6e6-514e-458d-840e-415c21f14bc8-kube-api-access-wzpxl\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:33:08.088388 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:08.088283    2576 generic.go:356] "Generic (PLEG): container finished" podID="078cd6e6-514e-458d-840e-415c21f14bc8" containerID="fdb01ca240939349e432b9f254e2f977bf41ca7a997dabd2743059ad6b7200d8" exitCode=0
Sep 11 18:33:08.088388 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:08.088356    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8" event={"ID":"078cd6e6-514e-458d-840e-415c21f14bc8","Type":"ContainerDied","Data":"fdb01ca240939349e432b9f254e2f977bf41ca7a997dabd2743059ad6b7200d8"}
Sep 11 18:33:08.088625 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:08.088366    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8"
Sep 11 18:33:08.088625 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:08.088421    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8" event={"ID":"078cd6e6-514e-458d-840e-415c21f14bc8","Type":"ContainerDied","Data":"7424c31a9cefe458030f805dc9eca06a1ddac3f088e85f854adacf4afe5bd045"}
Sep 11 18:33:08.088625 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:08.088434    2576 scope.go:117] "RemoveContainer" containerID="fdb01ca240939349e432b9f254e2f977bf41ca7a997dabd2743059ad6b7200d8"
Sep 11 18:33:08.089846 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:08.089817    2576 generic.go:356] "Generic (PLEG): container finished" podID="281d75e1-25ac-4c52-bad4-14f88222061e" containerID="ee2832db80af42390d65ac01d8a85a618f6c3185aa561234cb889ca2b30159e6" exitCode=0
Sep 11 18:33:08.089946 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:08.089856    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4"
Sep 11 18:33:08.089946 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:08.089897    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4" event={"ID":"281d75e1-25ac-4c52-bad4-14f88222061e","Type":"ContainerDied","Data":"ee2832db80af42390d65ac01d8a85a618f6c3185aa561234cb889ca2b30159e6"}
Sep 11 18:33:08.090045 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:08.089936    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4" event={"ID":"281d75e1-25ac-4c52-bad4-14f88222061e","Type":"ContainerDied","Data":"09074c67edf5936aa00004ef101eeda889a0198b112785f90e705fcd9922e6b0"}
Sep 11 18:33:08.099473 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:08.099447    2576 scope.go:117] "RemoveContainer" containerID="fdb01ca240939349e432b9f254e2f977bf41ca7a997dabd2743059ad6b7200d8"
Sep 11 18:33:08.099900 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:33:08.099869    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdb01ca240939349e432b9f254e2f977bf41ca7a997dabd2743059ad6b7200d8\": container with ID starting with fdb01ca240939349e432b9f254e2f977bf41ca7a997dabd2743059ad6b7200d8 not found: ID does not exist" containerID="fdb01ca240939349e432b9f254e2f977bf41ca7a997dabd2743059ad6b7200d8"
Sep 11 18:33:08.099986 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:08.099912    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdb01ca240939349e432b9f254e2f977bf41ca7a997dabd2743059ad6b7200d8"} err="failed to get container status \"fdb01ca240939349e432b9f254e2f977bf41ca7a997dabd2743059ad6b7200d8\": rpc error: code = NotFound desc = could not find container \"fdb01ca240939349e432b9f254e2f977bf41ca7a997dabd2743059ad6b7200d8\": container with ID starting with fdb01ca240939349e432b9f254e2f977bf41ca7a997dabd2743059ad6b7200d8 not found: ID does not exist"
Sep 11 18:33:08.099986 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:08.099936    2576 scope.go:117] "RemoveContainer" containerID="ee2832db80af42390d65ac01d8a85a618f6c3185aa561234cb889ca2b30159e6"
Sep 11 18:33:08.104447 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:08.104423    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:33:08.109819 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:08.109793    2576 scope.go:117] "RemoveContainer" containerID="ee2832db80af42390d65ac01d8a85a618f6c3185aa561234cb889ca2b30159e6"
Sep 11 18:33:08.110155 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:33:08.110135    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee2832db80af42390d65ac01d8a85a618f6c3185aa561234cb889ca2b30159e6\": container with ID starting with ee2832db80af42390d65ac01d8a85a618f6c3185aa561234cb889ca2b30159e6 not found: ID does not exist" containerID="ee2832db80af42390d65ac01d8a85a618f6c3185aa561234cb889ca2b30159e6"
Sep 11 18:33:08.110219 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:08.110165    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee2832db80af42390d65ac01d8a85a618f6c3185aa561234cb889ca2b30159e6"} err="failed to get container status \"ee2832db80af42390d65ac01d8a85a618f6c3185aa561234cb889ca2b30159e6\": rpc error: code = NotFound desc = could not find container \"ee2832db80af42390d65ac01d8a85a618f6c3185aa561234cb889ca2b30159e6\": container with ID starting with ee2832db80af42390d65ac01d8a85a618f6c3185aa561234cb889ca2b30159e6 not found: ID does not exist"
Sep 11 18:33:08.111501 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:08.111481    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8"]
Sep 11 18:33:08.115650 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:08.115624    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3e24a782-66c756b897-tf7g8"]
Sep 11 18:33:08.155842 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:08.148960    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4"]
Sep 11 18:33:08.158297 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:08.158194    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3e24a782-748755bf4b-4cns4"]
Sep 11 18:33:09.912399 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:09.912357    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="078cd6e6-514e-458d-840e-415c21f14bc8" path="/var/lib/kubelet/pods/078cd6e6-514e-458d-840e-415c21f14bc8/volumes"
Sep 11 18:33:09.912931 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:09.912911    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="281d75e1-25ac-4c52-bad4-14f88222061e" path="/var/lib/kubelet/pods/281d75e1-25ac-4c52-bad4-14f88222061e/volumes"
Sep 11 18:33:15.750888 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:15.750850    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"]
Sep 11 18:33:30.770658 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:30.770596    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5" podUID="f9ae30e6-f378-45b5-b382-d5e7568c46e0" containerName="main" containerID="cri-o://6b75a1ce01b4a7ea0719db37b575643779e1e7083e979ef3ddc025a27a5fad3f" gracePeriod=45
Sep 11 18:33:31.031171 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.031100    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:33:31.046956 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.046919    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9ae30e6-f378-45b5-b382-d5e7568c46e0-home\") pod \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\" (UID: \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\") "
Sep 11 18:33:31.046956 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.046962    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9ae30e6-f378-45b5-b382-d5e7568c46e0-model-cache\") pod \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\" (UID: \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\") "
Sep 11 18:33:31.047194 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.046990    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6txxl\" (UniqueName: \"kubernetes.io/projected/f9ae30e6-f378-45b5-b382-d5e7568c46e0-kube-api-access-6txxl\") pod \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\" (UID: \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\") "
Sep 11 18:33:31.047194 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.047012    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9ae30e6-f378-45b5-b382-d5e7568c46e0-dshm\") pod \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\" (UID: \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\") "
Sep 11 18:33:31.047194 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.047051    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9ae30e6-f378-45b5-b382-d5e7568c46e0-tmp-dir\") pod \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\" (UID: \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\") "
Sep 11 18:33:31.047194 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.047114    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9ae30e6-f378-45b5-b382-d5e7568c46e0-tls-certs\") pod \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\" (UID: \"f9ae30e6-f378-45b5-b382-d5e7568c46e0\") "
Sep 11 18:33:31.047481 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.047252    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ae30e6-f378-45b5-b382-d5e7568c46e0-home" (OuterVolumeSpecName: "home") pod "f9ae30e6-f378-45b5-b382-d5e7568c46e0" (UID: "f9ae30e6-f378-45b5-b382-d5e7568c46e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:33:31.047481 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.047260    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ae30e6-f378-45b5-b382-d5e7568c46e0-model-cache" (OuterVolumeSpecName: "model-cache") pod "f9ae30e6-f378-45b5-b382-d5e7568c46e0" (UID: "f9ae30e6-f378-45b5-b382-d5e7568c46e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:33:31.047481 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.047394    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9ae30e6-f378-45b5-b382-d5e7568c46e0-home\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:33:31.047481 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.047415    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9ae30e6-f378-45b5-b382-d5e7568c46e0-model-cache\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:33:31.047875 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.047606    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ae30e6-f378-45b5-b382-d5e7568c46e0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f9ae30e6-f378-45b5-b382-d5e7568c46e0" (UID: "f9ae30e6-f378-45b5-b382-d5e7568c46e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:33:31.050572 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.050418    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9ae30e6-f378-45b5-b382-d5e7568c46e0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f9ae30e6-f378-45b5-b382-d5e7568c46e0" (UID: "f9ae30e6-f378-45b5-b382-d5e7568c46e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:33:31.050572 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.050525    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ae30e6-f378-45b5-b382-d5e7568c46e0-kube-api-access-6txxl" (OuterVolumeSpecName: "kube-api-access-6txxl") pod "f9ae30e6-f378-45b5-b382-d5e7568c46e0" (UID: "f9ae30e6-f378-45b5-b382-d5e7568c46e0"). InnerVolumeSpecName "kube-api-access-6txxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:33:31.050572 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.050561    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ae30e6-f378-45b5-b382-d5e7568c46e0-dshm" (OuterVolumeSpecName: "dshm") pod "f9ae30e6-f378-45b5-b382-d5e7568c46e0" (UID: "f9ae30e6-f378-45b5-b382-d5e7568c46e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:33:31.148222 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.148187    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6txxl\" (UniqueName: \"kubernetes.io/projected/f9ae30e6-f378-45b5-b382-d5e7568c46e0-kube-api-access-6txxl\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:33:31.148222 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.148218    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9ae30e6-f378-45b5-b382-d5e7568c46e0-dshm\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:33:31.148222 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.148229    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9ae30e6-f378-45b5-b382-d5e7568c46e0-tmp-dir\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:33:31.148460 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.148238    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9ae30e6-f378-45b5-b382-d5e7568c46e0-tls-certs\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:33:31.214543 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.214493    2576 generic.go:356] "Generic (PLEG): container finished" podID="f9ae30e6-f378-45b5-b382-d5e7568c46e0" containerID="6b75a1ce01b4a7ea0719db37b575643779e1e7083e979ef3ddc025a27a5fad3f" exitCode=0
Sep 11 18:33:31.214543 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.214537    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"
Sep 11 18:33:31.214772 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.214575    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5" event={"ID":"f9ae30e6-f378-45b5-b382-d5e7568c46e0","Type":"ContainerDied","Data":"6b75a1ce01b4a7ea0719db37b575643779e1e7083e979ef3ddc025a27a5fad3f"}
Sep 11 18:33:31.214772 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.214637    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5" event={"ID":"f9ae30e6-f378-45b5-b382-d5e7568c46e0","Type":"ContainerDied","Data":"55d3fbc46537f2691992d3695dd75035ffaeedaaf48a237d88191e1d07d558a5"}
Sep 11 18:33:31.214772 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.214657    2576 scope.go:117] "RemoveContainer" containerID="6b75a1ce01b4a7ea0719db37b575643779e1e7083e979ef3ddc025a27a5fad3f"
Sep 11 18:33:31.223832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.223811    2576 scope.go:117] "RemoveContainer" containerID="6b75a1ce01b4a7ea0719db37b575643779e1e7083e979ef3ddc025a27a5fad3f"
Sep 11 18:33:31.224121 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:33:31.224105    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b75a1ce01b4a7ea0719db37b575643779e1e7083e979ef3ddc025a27a5fad3f\": container with ID starting with 6b75a1ce01b4a7ea0719db37b575643779e1e7083e979ef3ddc025a27a5fad3f not found: ID does not exist" containerID="6b75a1ce01b4a7ea0719db37b575643779e1e7083e979ef3ddc025a27a5fad3f"
Sep 11 18:33:31.224179 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.224129    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b75a1ce01b4a7ea0719db37b575643779e1e7083e979ef3ddc025a27a5fad3f"} err="failed to get container status \"6b75a1ce01b4a7ea0719db37b575643779e1e7083e979ef3ddc025a27a5fad3f\": rpc error: code = NotFound desc = could not find container \"6b75a1ce01b4a7ea0719db37b575643779e1e7083e979ef3ddc025a27a5fad3f\": container with ID starting with 6b75a1ce01b4a7ea0719db37b575643779e1e7083e979ef3ddc025a27a5fad3f not found: ID does not exist"
Sep 11 18:33:31.234982 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.234953    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"]
Sep 11 18:33:31.238725 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.238699    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3e24a782/e2e-lifecycle-model-3e24a782-kserve-6cf4bfbbc5-fmhn5"]
Sep 11 18:33:31.910457 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:31.910419    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9ae30e6-f378-45b5-b382-d5e7568c46e0" path="/var/lib/kubelet/pods/f9ae30e6-f378-45b5-b382-d5e7568c46e0/volumes"
Sep 11 18:33:36.103934 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:36.103902    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:33:41.697146 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.697108    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"]
Sep 11 18:33:41.697515 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.697460    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="281d75e1-25ac-4c52-bad4-14f88222061e" containerName="payload-pre-processing"
Sep 11 18:33:41.697515 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.697470    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="281d75e1-25ac-4c52-bad4-14f88222061e" containerName="payload-pre-processing"
Sep 11 18:33:41.697515 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.697489    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9ae30e6-f378-45b5-b382-d5e7568c46e0" containerName="main"
Sep 11 18:33:41.697515 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.697495    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9ae30e6-f378-45b5-b382-d5e7568c46e0" containerName="main"
Sep 11 18:33:41.697515 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.697507    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="078cd6e6-514e-458d-840e-415c21f14bc8" containerName="payload-processing"
Sep 11 18:33:41.697515 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.697513    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="078cd6e6-514e-458d-840e-415c21f14bc8" containerName="payload-processing"
Sep 11 18:33:41.697731 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.697604    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="281d75e1-25ac-4c52-bad4-14f88222061e" containerName="payload-pre-processing"
Sep 11 18:33:41.697731 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.697617    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="078cd6e6-514e-458d-840e-415c21f14bc8" containerName="payload-processing"
Sep 11 18:33:41.697731 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.697625    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9ae30e6-f378-45b5-b382-d5e7568c46e0" containerName="main"
Sep 11 18:33:41.700569 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.700552    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:33:41.702706 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.702685    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-553596\"/\"distinct-2-w4-553596-kserve-self-signed-certs\""
Sep 11 18:33:41.710808 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.710785    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"]
Sep 11 18:33:41.742462 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.742429    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cwf8\" (UniqueName: \"kubernetes.io/projected/ec02549b-7498-46de-86f6-9b7ea8b2c846-kube-api-access-4cwf8\") pod \"distinct-2-w4-553596-kserve-cc4bb45b9-2kz64\" (UID: \"ec02549b-7498-46de-86f6-9b7ea8b2c846\") " pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:33:41.742618 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.742480    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec02549b-7498-46de-86f6-9b7ea8b2c846-tmp-dir\") pod \"distinct-2-w4-553596-kserve-cc4bb45b9-2kz64\" (UID: \"ec02549b-7498-46de-86f6-9b7ea8b2c846\") " pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:33:41.742618 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.742510    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec02549b-7498-46de-86f6-9b7ea8b2c846-model-cache\") pod \"distinct-2-w4-553596-kserve-cc4bb45b9-2kz64\" (UID: \"ec02549b-7498-46de-86f6-9b7ea8b2c846\") " pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:33:41.742618 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.742543    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec02549b-7498-46de-86f6-9b7ea8b2c846-dshm\") pod \"distinct-2-w4-553596-kserve-cc4bb45b9-2kz64\" (UID: \"ec02549b-7498-46de-86f6-9b7ea8b2c846\") " pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:33:41.742618 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.742561    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec02549b-7498-46de-86f6-9b7ea8b2c846-tls-certs\") pod \"distinct-2-w4-553596-kserve-cc4bb45b9-2kz64\" (UID: \"ec02549b-7498-46de-86f6-9b7ea8b2c846\") " pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:33:41.742765 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.742660    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec02549b-7498-46de-86f6-9b7ea8b2c846-home\") pod \"distinct-2-w4-553596-kserve-cc4bb45b9-2kz64\" (UID: \"ec02549b-7498-46de-86f6-9b7ea8b2c846\") " pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:33:41.843478 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.843439    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec02549b-7498-46de-86f6-9b7ea8b2c846-dshm\") pod \"distinct-2-w4-553596-kserve-cc4bb45b9-2kz64\" (UID: \"ec02549b-7498-46de-86f6-9b7ea8b2c846\") " pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:33:41.843478 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.843477    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec02549b-7498-46de-86f6-9b7ea8b2c846-tls-certs\") pod \"distinct-2-w4-553596-kserve-cc4bb45b9-2kz64\" (UID: \"ec02549b-7498-46de-86f6-9b7ea8b2c846\") " pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:33:41.843753 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.843634    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec02549b-7498-46de-86f6-9b7ea8b2c846-home\") pod \"distinct-2-w4-553596-kserve-cc4bb45b9-2kz64\" (UID: \"ec02549b-7498-46de-86f6-9b7ea8b2c846\") " pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:33:41.843753 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.843737    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cwf8\" (UniqueName: \"kubernetes.io/projected/ec02549b-7498-46de-86f6-9b7ea8b2c846-kube-api-access-4cwf8\") pod \"distinct-2-w4-553596-kserve-cc4bb45b9-2kz64\" (UID: \"ec02549b-7498-46de-86f6-9b7ea8b2c846\") " pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:33:41.843863 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.843832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec02549b-7498-46de-86f6-9b7ea8b2c846-tmp-dir\") pod \"distinct-2-w4-553596-kserve-cc4bb45b9-2kz64\" (UID: \"ec02549b-7498-46de-86f6-9b7ea8b2c846\") " pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:33:41.843924 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.843867    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec02549b-7498-46de-86f6-9b7ea8b2c846-model-cache\") pod \"distinct-2-w4-553596-kserve-cc4bb45b9-2kz64\" (UID: \"ec02549b-7498-46de-86f6-9b7ea8b2c846\") " pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:33:41.844097 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.844071    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec02549b-7498-46de-86f6-9b7ea8b2c846-home\") pod \"distinct-2-w4-553596-kserve-cc4bb45b9-2kz64\" (UID: \"ec02549b-7498-46de-86f6-9b7ea8b2c846\") " pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:33:41.844228 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.844126    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec02549b-7498-46de-86f6-9b7ea8b2c846-tmp-dir\") pod \"distinct-2-w4-553596-kserve-cc4bb45b9-2kz64\" (UID: \"ec02549b-7498-46de-86f6-9b7ea8b2c846\") " pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:33:41.844228 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.844193    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec02549b-7498-46de-86f6-9b7ea8b2c846-model-cache\") pod \"distinct-2-w4-553596-kserve-cc4bb45b9-2kz64\" (UID: \"ec02549b-7498-46de-86f6-9b7ea8b2c846\") " pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:33:41.845820 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.845800    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec02549b-7498-46de-86f6-9b7ea8b2c846-dshm\") pod \"distinct-2-w4-553596-kserve-cc4bb45b9-2kz64\" (UID: \"ec02549b-7498-46de-86f6-9b7ea8b2c846\") " pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:33:41.846096 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.846077    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec02549b-7498-46de-86f6-9b7ea8b2c846-tls-certs\") pod \"distinct-2-w4-553596-kserve-cc4bb45b9-2kz64\" (UID: \"ec02549b-7498-46de-86f6-9b7ea8b2c846\") " pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:33:41.850688 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:41.850670    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cwf8\" (UniqueName: \"kubernetes.io/projected/ec02549b-7498-46de-86f6-9b7ea8b2c846-kube-api-access-4cwf8\") pod \"distinct-2-w4-553596-kserve-cc4bb45b9-2kz64\" (UID: \"ec02549b-7498-46de-86f6-9b7ea8b2c846\") " pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:33:42.009856 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:42.009765    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:33:42.279637 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:42.279611    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"]
Sep 11 18:33:42.281747 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:33:42.281711    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec02549b_7498_46de_86f6_9b7ea8b2c846.slice/crio-6575ec84716bcdb2a6f00f2583b427ddb3f77916b914f5d60aaa96e59d480270 WatchSource:0}: Error finding container 6575ec84716bcdb2a6f00f2583b427ddb3f77916b914f5d60aaa96e59d480270: Status 404 returned error can't find the container with id 6575ec84716bcdb2a6f00f2583b427ddb3f77916b914f5d60aaa96e59d480270
Sep 11 18:33:43.274920 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:43.274878    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64" event={"ID":"ec02549b-7498-46de-86f6-9b7ea8b2c846","Type":"ContainerStarted","Data":"82766962126faad71f1baf3408f9236ffe4d0d4c2af23d424162330d1767241b"}
Sep 11 18:33:43.274920 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:43.274924    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64" event={"ID":"ec02549b-7498-46de-86f6-9b7ea8b2c846","Type":"ContainerStarted","Data":"6575ec84716bcdb2a6f00f2583b427ddb3f77916b914f5d60aaa96e59d480270"}
Sep 11 18:33:43.293093 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:43.293046    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64" podStartSLOduration=2.293031788 podStartE2EDuration="2.293031788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:33:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:33:43.29115741 +0000 UTC m=+1421.892518575" watchObservedRunningTime="2026-09-11 18:33:43.293031788 +0000 UTC m=+1421.894392994"
Sep 11 18:33:52.010485 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:52.010434    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:33:52.010930 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:52.010497    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:33:52.023701 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:52.023669    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:33:52.332784 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:33:52.332691    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:34:07.497390 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:07.497351    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"]
Sep 11 18:34:07.502202 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:07.502173    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:34:07.504776 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:07.504739    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d73031\"/\"embedding-w0-d73031-kserve-self-signed-certs\""
Sep 11 18:34:07.511625 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:07.511593    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"]
Sep 11 18:34:07.591718 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:07.591667    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/145b5179-374e-498b-a7ef-1a70201eb4af-tmp-dir\") pod \"embedding-w0-d73031-kserve-5764dbf97d-5k56f\" (UID: \"145b5179-374e-498b-a7ef-1a70201eb4af\") " pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:34:07.591718 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:07.591729    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/145b5179-374e-498b-a7ef-1a70201eb4af-tls-certs\") pod \"embedding-w0-d73031-kserve-5764dbf97d-5k56f\" (UID: \"145b5179-374e-498b-a7ef-1a70201eb4af\") " pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:34:07.591940 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:07.591799    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/145b5179-374e-498b-a7ef-1a70201eb4af-model-cache\") pod \"embedding-w0-d73031-kserve-5764dbf97d-5k56f\" (UID: \"145b5179-374e-498b-a7ef-1a70201eb4af\") " pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:34:07.591940 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:07.591847    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/145b5179-374e-498b-a7ef-1a70201eb4af-home\") pod \"embedding-w0-d73031-kserve-5764dbf97d-5k56f\" (UID: \"145b5179-374e-498b-a7ef-1a70201eb4af\") " pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:34:07.592019 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:07.591939    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx2c7\" (UniqueName: \"kubernetes.io/projected/145b5179-374e-498b-a7ef-1a70201eb4af-kube-api-access-mx2c7\") pod \"embedding-w0-d73031-kserve-5764dbf97d-5k56f\" (UID: \"145b5179-374e-498b-a7ef-1a70201eb4af\") " pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:34:07.592019 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:07.591985    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/145b5179-374e-498b-a7ef-1a70201eb4af-dshm\") pod \"embedding-w0-d73031-kserve-5764dbf97d-5k56f\" (UID: \"145b5179-374e-498b-a7ef-1a70201eb4af\") " pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:34:07.693010 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:07.692965    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx2c7\" (UniqueName: \"kubernetes.io/projected/145b5179-374e-498b-a7ef-1a70201eb4af-kube-api-access-mx2c7\") pod \"embedding-w0-d73031-kserve-5764dbf97d-5k56f\" (UID: \"145b5179-374e-498b-a7ef-1a70201eb4af\") " pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:34:07.693010 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:07.693018    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/145b5179-374e-498b-a7ef-1a70201eb4af-dshm\") pod \"embedding-w0-d73031-kserve-5764dbf97d-5k56f\" (UID: \"145b5179-374e-498b-a7ef-1a70201eb4af\") " pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:34:07.693311 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:07.693045    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/145b5179-374e-498b-a7ef-1a70201eb4af-tmp-dir\") pod \"embedding-w0-d73031-kserve-5764dbf97d-5k56f\" (UID: \"145b5179-374e-498b-a7ef-1a70201eb4af\") " pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:34:07.693311 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:07.693077    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/145b5179-374e-498b-a7ef-1a70201eb4af-tls-certs\") pod \"embedding-w0-d73031-kserve-5764dbf97d-5k56f\" (UID: \"145b5179-374e-498b-a7ef-1a70201eb4af\") " pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:34:07.693311 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:07.693099    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/145b5179-374e-498b-a7ef-1a70201eb4af-model-cache\") pod \"embedding-w0-d73031-kserve-5764dbf97d-5k56f\" (UID: \"145b5179-374e-498b-a7ef-1a70201eb4af\") " pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:34:07.693311 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:07.693123    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/145b5179-374e-498b-a7ef-1a70201eb4af-home\") pod \"embedding-w0-d73031-kserve-5764dbf97d-5k56f\" (UID: \"145b5179-374e-498b-a7ef-1a70201eb4af\") " pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:34:07.693515 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:07.693472    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/145b5179-374e-498b-a7ef-1a70201eb4af-home\") pod \"embedding-w0-d73031-kserve-5764dbf97d-5k56f\" (UID: \"145b5179-374e-498b-a7ef-1a70201eb4af\") " pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:34:07.693515 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:07.693503    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/145b5179-374e-498b-a7ef-1a70201eb4af-tmp-dir\") pod \"embedding-w0-d73031-kserve-5764dbf97d-5k56f\" (UID: \"145b5179-374e-498b-a7ef-1a70201eb4af\") " pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:34:07.693661 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:07.693642    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/145b5179-374e-498b-a7ef-1a70201eb4af-model-cache\") pod \"embedding-w0-d73031-kserve-5764dbf97d-5k56f\" (UID: \"145b5179-374e-498b-a7ef-1a70201eb4af\") " pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:34:07.695661 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:07.695639    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/145b5179-374e-498b-a7ef-1a70201eb4af-dshm\") pod \"embedding-w0-d73031-kserve-5764dbf97d-5k56f\" (UID: \"145b5179-374e-498b-a7ef-1a70201eb4af\") " pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:34:07.695883 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:07.695864    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/145b5179-374e-498b-a7ef-1a70201eb4af-tls-certs\") pod \"embedding-w0-d73031-kserve-5764dbf97d-5k56f\" (UID: \"145b5179-374e-498b-a7ef-1a70201eb4af\") " pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:34:07.701327 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:07.701299    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx2c7\" (UniqueName: \"kubernetes.io/projected/145b5179-374e-498b-a7ef-1a70201eb4af-kube-api-access-mx2c7\") pod \"embedding-w0-d73031-kserve-5764dbf97d-5k56f\" (UID: \"145b5179-374e-498b-a7ef-1a70201eb4af\") " pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:34:07.813357 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:07.813256    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:34:08.088048 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:08.088015    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"]
Sep 11 18:34:08.090650 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:34:08.090617    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod145b5179_374e_498b_a7ef_1a70201eb4af.slice/crio-b7865b57ada579e49a550d8fa8991faa73cdfc591a941dc56da10d9b091f6a6b WatchSource:0}: Error finding container b7865b57ada579e49a550d8fa8991faa73cdfc591a941dc56da10d9b091f6a6b: Status 404 returned error can't find the container with id b7865b57ada579e49a550d8fa8991faa73cdfc591a941dc56da10d9b091f6a6b
Sep 11 18:34:08.412421 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:08.412318    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f" event={"ID":"145b5179-374e-498b-a7ef-1a70201eb4af","Type":"ContainerStarted","Data":"c4d9ffbf3da9990e890e3b60092b42bfbe1dd90a14afe7cfcbfa4a83f68a1068"}
Sep 11 18:34:08.412421 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:08.412364    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f" event={"ID":"145b5179-374e-498b-a7ef-1a70201eb4af","Type":"ContainerStarted","Data":"b7865b57ada579e49a550d8fa8991faa73cdfc591a941dc56da10d9b091f6a6b"}
Sep 11 18:34:08.435752 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:08.435693    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f" podStartSLOduration=1.435672311 podStartE2EDuration="1.435672311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:34:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:34:08.431853833 +0000 UTC m=+1447.033214996" watchObservedRunningTime="2026-09-11 18:34:08.435672311 +0000 UTC m=+1447.037033474"
Sep 11 18:34:11.216963 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:11.216917    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:34:17.813743 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:17.813698    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:34:17.814145 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:17.813760    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:34:17.826519 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:17.826484    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:34:18.477335 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:18.477304    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:34:46.024849 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:34:46.024814    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:35:07.536113 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:07.536077    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d73031-s2t5b"]
Sep 11 18:35:07.539782 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:07.539761    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d73031-s2t5b"
Sep 11 18:35:07.542109 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:07.542084    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t9kdx\""
Sep 11 18:35:07.548100 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:07.548065    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d73031-s2t5b"]
Sep 11 18:35:07.637458 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:07.637418    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0684132b-dd70-41e1-8d84-45335355805f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d73031-s2t5b\" (UID: \"0684132b-dd70-41e1-8d84-45335355805f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d73031-s2t5b"
Sep 11 18:35:07.738029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:07.737985    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0684132b-dd70-41e1-8d84-45335355805f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d73031-s2t5b\" (UID: \"0684132b-dd70-41e1-8d84-45335355805f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d73031-s2t5b"
Sep 11 18:35:07.738728 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:07.738704    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0684132b-dd70-41e1-8d84-45335355805f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d73031-s2t5b\" (UID: \"0684132b-dd70-41e1-8d84-45335355805f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d73031-s2t5b"
Sep 11 18:35:07.853361 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:07.853272    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d73031-s2t5b"
Sep 11 18:35:08.124594 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:08.124547    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d73031-s2t5b"]
Sep 11 18:35:08.126695 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:35:08.126669    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0684132b_dd70_41e1_8d84_45335355805f.slice/crio-1edc9feee3ca48d089ebb046d30fa5ea41bb2f5307f4b8c6a06c3ec924752400 WatchSource:0}: Error finding container 1edc9feee3ca48d089ebb046d30fa5ea41bb2f5307f4b8c6a06c3ec924752400: Status 404 returned error can't find the container with id 1edc9feee3ca48d089ebb046d30fa5ea41bb2f5307f4b8c6a06c3ec924752400
Sep 11 18:35:08.721847 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:08.721753    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d73031-s2t5b" event={"ID":"0684132b-dd70-41e1-8d84-45335355805f","Type":"ContainerStarted","Data":"81a71313a494b18377c083a6e9329ea9b3c2cb9a2da626a53a10cc7bb321651c"}
Sep 11 18:35:08.721847 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:08.721804    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d73031-s2t5b" event={"ID":"0684132b-dd70-41e1-8d84-45335355805f","Type":"ContainerStarted","Data":"1edc9feee3ca48d089ebb046d30fa5ea41bb2f5307f4b8c6a06c3ec924752400"}
Sep 11 18:35:08.737522 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:08.737454    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d73031-s2t5b" podStartSLOduration=1.737436354 podStartE2EDuration="1.737436354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:35:08.736180606 +0000 UTC m=+1507.337541787" watchObservedRunningTime="2026-09-11 18:35:08.737436354 +0000 UTC m=+1507.338797518"
Sep 11 18:35:09.728448 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:09.728409    2576 generic.go:356] "Generic (PLEG): container finished" podID="0684132b-dd70-41e1-8d84-45335355805f" containerID="81a71313a494b18377c083a6e9329ea9b3c2cb9a2da626a53a10cc7bb321651c" exitCode=0
Sep 11 18:35:09.728928 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:09.728495    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d73031-s2t5b" event={"ID":"0684132b-dd70-41e1-8d84-45335355805f","Type":"ContainerDied","Data":"81a71313a494b18377c083a6e9329ea9b3c2cb9a2da626a53a10cc7bb321651c"}
Sep 11 18:35:10.875019 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:10.874993    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d73031-s2t5b"
Sep 11 18:35:10.971415 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:10.971380    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0684132b-dd70-41e1-8d84-45335355805f-maas-api-service-ca\") pod \"0684132b-dd70-41e1-8d84-45335355805f\" (UID: \"0684132b-dd70-41e1-8d84-45335355805f\") "
Sep 11 18:35:10.971840 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:10.971819    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0684132b-dd70-41e1-8d84-45335355805f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0684132b-dd70-41e1-8d84-45335355805f" (UID: "0684132b-dd70-41e1-8d84-45335355805f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:35:11.006127 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:11.006049    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t"]
Sep 11 18:35:11.006453 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:11.006441    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0684132b-dd70-41e1-8d84-45335355805f" containerName="revoke-keys"
Sep 11 18:35:11.006521 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:11.006454    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0684132b-dd70-41e1-8d84-45335355805f" containerName="revoke-keys"
Sep 11 18:35:11.006521 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:11.006515    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0684132b-dd70-41e1-8d84-45335355805f" containerName="revoke-keys"
Sep 11 18:35:11.009558 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:11.009537    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t"
Sep 11 18:35:11.011771 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:11.011748    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1e370a\""
Sep 11 18:35:11.011771 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:11.011769    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1e370a-dockercfg-grqnn\""
Sep 11 18:35:11.020787 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:11.020760    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t"]
Sep 11 18:35:11.072357 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:11.072317    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0684132b-dd70-41e1-8d84-45335355805f-maas-api-service-ca\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:35:11.173735 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:11.173686    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a4b88d-51fa-4dc4-8359-65ac102c172a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t\" (UID: \"70a4b88d-51fa-4dc4-8359-65ac102c172a\") " pod="openshift-ingress/payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t"
Sep 11 18:35:11.173942 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:11.173833    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85m7v\" (UniqueName: \"kubernetes.io/projected/70a4b88d-51fa-4dc4-8359-65ac102c172a-kube-api-access-85m7v\") pod \"payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t\" (UID: \"70a4b88d-51fa-4dc4-8359-65ac102c172a\") " pod="openshift-ingress/payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t"
Sep 11 18:35:11.274876 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:11.274786    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a4b88d-51fa-4dc4-8359-65ac102c172a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t\" (UID: \"70a4b88d-51fa-4dc4-8359-65ac102c172a\") " pod="openshift-ingress/payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t"
Sep 11 18:35:11.275113 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:11.274879    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85m7v\" (UniqueName: \"kubernetes.io/projected/70a4b88d-51fa-4dc4-8359-65ac102c172a-kube-api-access-85m7v\") pod \"payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t\" (UID: \"70a4b88d-51fa-4dc4-8359-65ac102c172a\") " pod="openshift-ingress/payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t"
Sep 11 18:35:11.275541 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:11.275518    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a4b88d-51fa-4dc4-8359-65ac102c172a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t\" (UID: \"70a4b88d-51fa-4dc4-8359-65ac102c172a\") " pod="openshift-ingress/payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t"
Sep 11 18:35:11.283512 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:11.283478    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85m7v\" (UniqueName: \"kubernetes.io/projected/70a4b88d-51fa-4dc4-8359-65ac102c172a-kube-api-access-85m7v\") pod \"payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t\" (UID: \"70a4b88d-51fa-4dc4-8359-65ac102c172a\") " pod="openshift-ingress/payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t"
Sep 11 18:35:11.321134 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:11.321093    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t"
Sep 11 18:35:11.594835 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:11.594805    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t"]
Sep 11 18:35:11.597226 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:35:11.597200    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a4b88d_51fa_4dc4_8359_65ac102c172a.slice/crio-63bcf42639d34017ae9853ede71dd7e052fe8f2ce4ab0c59eddfe267950eb2b5 WatchSource:0}: Error finding container 63bcf42639d34017ae9853ede71dd7e052fe8f2ce4ab0c59eddfe267950eb2b5: Status 404 returned error can't find the container with id 63bcf42639d34017ae9853ede71dd7e052fe8f2ce4ab0c59eddfe267950eb2b5
Sep 11 18:35:11.741154 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:11.741115    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d73031-s2t5b"
Sep 11 18:35:11.741354 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:11.741122    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d73031-s2t5b" event={"ID":"0684132b-dd70-41e1-8d84-45335355805f","Type":"ContainerDied","Data":"1edc9feee3ca48d089ebb046d30fa5ea41bb2f5307f4b8c6a06c3ec924752400"}
Sep 11 18:35:11.741354 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:11.741272    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1edc9feee3ca48d089ebb046d30fa5ea41bb2f5307f4b8c6a06c3ec924752400"
Sep 11 18:35:11.746609 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:11.743560    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t" event={"ID":"70a4b88d-51fa-4dc4-8359-65ac102c172a","Type":"ContainerStarted","Data":"56d2b91f83fbfe5044cebc2056037ec2787e79c6299cc216c709741c13f03dc9"}
Sep 11 18:35:11.746609 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:11.743643    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t" event={"ID":"70a4b88d-51fa-4dc4-8359-65ac102c172a","Type":"ContainerStarted","Data":"63bcf42639d34017ae9853ede71dd7e052fe8f2ce4ab0c59eddfe267950eb2b5"}
Sep 11 18:35:11.746609 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:11.744444    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t"
Sep 11 18:35:11.772534 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:11.768504    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t" podStartSLOduration=1.7684820860000001 podStartE2EDuration="1.768482086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:35:11.764709202 +0000 UTC m=+1510.366070366" watchObservedRunningTime="2026-09-11 18:35:11.768482086 +0000 UTC m=+1510.369843253"
Sep 11 18:35:12.406201 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:12.406164    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"]
Sep 11 18:35:12.415039 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:12.415008    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"]
Sep 11 18:35:15.911872 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:15.911837    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:35:23.365016 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:23.364984    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2"]
Sep 11 18:35:23.365845 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:23.365783    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2" podUID="13435514-2ef4-401b-b2d6-cfe8fd0f52ef" containerName="payload-pre-processing" containerID="cri-o://c93fae0d29061d82eb009271eaa4ed6b9e2650af11e9d6bb72b935d3eb7622db" gracePeriod=30
Sep 11 18:35:23.530281 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:23.530245    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d73031-s2t5b"]
Sep 11 18:35:23.532126 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:23.532103    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d73031-s2t5b"]
Sep 11 18:35:23.630442 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:23.630381    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2"
Sep 11 18:35:23.687671 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:23.687631    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljj5g\" (UniqueName: \"kubernetes.io/projected/13435514-2ef4-401b-b2d6-cfe8fd0f52ef-kube-api-access-ljj5g\") pod \"13435514-2ef4-401b-b2d6-cfe8fd0f52ef\" (UID: \"13435514-2ef4-401b-b2d6-cfe8fd0f52ef\") "
Sep 11 18:35:23.687867 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:23.687698    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13435514-2ef4-401b-b2d6-cfe8fd0f52ef-plugins-config-volume\") pod \"13435514-2ef4-401b-b2d6-cfe8fd0f52ef\" (UID: \"13435514-2ef4-401b-b2d6-cfe8fd0f52ef\") "
Sep 11 18:35:23.688104 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:23.688076    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13435514-2ef4-401b-b2d6-cfe8fd0f52ef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "13435514-2ef4-401b-b2d6-cfe8fd0f52ef" (UID: "13435514-2ef4-401b-b2d6-cfe8fd0f52ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:35:23.689811 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:23.689783    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13435514-2ef4-401b-b2d6-cfe8fd0f52ef-kube-api-access-ljj5g" (OuterVolumeSpecName: "kube-api-access-ljj5g") pod "13435514-2ef4-401b-b2d6-cfe8fd0f52ef" (UID: "13435514-2ef4-401b-b2d6-cfe8fd0f52ef"). InnerVolumeSpecName "kube-api-access-ljj5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:35:23.757340 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:23.757308    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t"
Sep 11 18:35:23.789548 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:23.789513    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljj5g\" (UniqueName: \"kubernetes.io/projected/13435514-2ef4-401b-b2d6-cfe8fd0f52ef-kube-api-access-ljj5g\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:35:23.789750 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:23.789627    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13435514-2ef4-401b-b2d6-cfe8fd0f52ef-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:35:23.814522 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:23.814490    2576 generic.go:356] "Generic (PLEG): container finished" podID="13435514-2ef4-401b-b2d6-cfe8fd0f52ef" containerID="c93fae0d29061d82eb009271eaa4ed6b9e2650af11e9d6bb72b935d3eb7622db" exitCode=0
Sep 11 18:35:23.814714 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:23.814557    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2"
Sep 11 18:35:23.814714 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:23.814561    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2" event={"ID":"13435514-2ef4-401b-b2d6-cfe8fd0f52ef","Type":"ContainerDied","Data":"c93fae0d29061d82eb009271eaa4ed6b9e2650af11e9d6bb72b935d3eb7622db"}
Sep 11 18:35:23.814714 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:23.814699    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2" event={"ID":"13435514-2ef4-401b-b2d6-cfe8fd0f52ef","Type":"ContainerDied","Data":"6818c90478e7732975cf2cddda5f7ba4ea25fbef68fd75c10a505042e59f4c73"}
Sep 11 18:35:23.814820 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:23.814722    2576 scope.go:117] "RemoveContainer" containerID="c93fae0d29061d82eb009271eaa4ed6b9e2650af11e9d6bb72b935d3eb7622db"
Sep 11 18:35:23.824071 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:23.824048    2576 scope.go:117] "RemoveContainer" containerID="c93fae0d29061d82eb009271eaa4ed6b9e2650af11e9d6bb72b935d3eb7622db"
Sep 11 18:35:23.824422 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:35:23.824401    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c93fae0d29061d82eb009271eaa4ed6b9e2650af11e9d6bb72b935d3eb7622db\": container with ID starting with c93fae0d29061d82eb009271eaa4ed6b9e2650af11e9d6bb72b935d3eb7622db not found: ID does not exist" containerID="c93fae0d29061d82eb009271eaa4ed6b9e2650af11e9d6bb72b935d3eb7622db"
Sep 11 18:35:23.824493 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:23.824430    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93fae0d29061d82eb009271eaa4ed6b9e2650af11e9d6bb72b935d3eb7622db"} err="failed to get container status \"c93fae0d29061d82eb009271eaa4ed6b9e2650af11e9d6bb72b935d3eb7622db\": rpc error: code = NotFound desc = could not find container \"c93fae0d29061d82eb009271eaa4ed6b9e2650af11e9d6bb72b935d3eb7622db\": container with ID starting with c93fae0d29061d82eb009271eaa4ed6b9e2650af11e9d6bb72b935d3eb7622db not found: ID does not exist"
Sep 11 18:35:23.838547 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:23.838513    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2"]
Sep 11 18:35:23.840708 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:23.840683    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d73031-5d5b5fd68b-msqq2"]
Sep 11 18:35:23.911572 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:23.911491    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0684132b-dd70-41e1-8d84-45335355805f" path="/var/lib/kubelet/pods/0684132b-dd70-41e1-8d84-45335355805f/volumes"
Sep 11 18:35:23.911888 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:23.911874    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13435514-2ef4-401b-b2d6-cfe8fd0f52ef" path="/var/lib/kubelet/pods/13435514-2ef4-401b-b2d6-cfe8fd0f52ef/volumes"
Sep 11 18:35:27.426733 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.426691    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n" podUID="c73362b7-f9eb-4789-8d84-062491a7dd45" containerName="main" containerID="cri-o://d8e424845dea51271bf5f59eb74799855bc3e7483118b21dc876a414b27b6e10" gracePeriod=45
Sep 11 18:35:27.439085 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.439042    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f" podUID="145b5179-374e-498b-a7ef-1a70201eb4af" containerName="main" containerID="cri-o://c4d9ffbf3da9990e890e3b60092b42bfbe1dd90a14afe7cfcbfa4a83f68a1068" gracePeriod=45
Sep 11 18:35:27.718778 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.718751    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:35:27.721392 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.721371    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:35:27.827369 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.827335    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/145b5179-374e-498b-a7ef-1a70201eb4af-model-cache\") pod \"145b5179-374e-498b-a7ef-1a70201eb4af\" (UID: \"145b5179-374e-498b-a7ef-1a70201eb4af\") "
Sep 11 18:35:27.827559 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.827381    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c73362b7-f9eb-4789-8d84-062491a7dd45-model-cache\") pod \"c73362b7-f9eb-4789-8d84-062491a7dd45\" (UID: \"c73362b7-f9eb-4789-8d84-062491a7dd45\") "
Sep 11 18:35:27.827559 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.827423    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/145b5179-374e-498b-a7ef-1a70201eb4af-tls-certs\") pod \"145b5179-374e-498b-a7ef-1a70201eb4af\" (UID: \"145b5179-374e-498b-a7ef-1a70201eb4af\") "
Sep 11 18:35:27.827559 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.827475    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c73362b7-f9eb-4789-8d84-062491a7dd45-dshm\") pod \"c73362b7-f9eb-4789-8d84-062491a7dd45\" (UID: \"c73362b7-f9eb-4789-8d84-062491a7dd45\") "
Sep 11 18:35:27.827559 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.827507    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/145b5179-374e-498b-a7ef-1a70201eb4af-home\") pod \"145b5179-374e-498b-a7ef-1a70201eb4af\" (UID: \"145b5179-374e-498b-a7ef-1a70201eb4af\") "
Sep 11 18:35:27.827813 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.827610    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/145b5179-374e-498b-a7ef-1a70201eb4af-model-cache" (OuterVolumeSpecName: "model-cache") pod "145b5179-374e-498b-a7ef-1a70201eb4af" (UID: "145b5179-374e-498b-a7ef-1a70201eb4af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:27.827813 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.827749    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73362b7-f9eb-4789-8d84-062491a7dd45-model-cache" (OuterVolumeSpecName: "model-cache") pod "c73362b7-f9eb-4789-8d84-062491a7dd45" (UID: "c73362b7-f9eb-4789-8d84-062491a7dd45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:27.827813 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.827784    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/145b5179-374e-498b-a7ef-1a70201eb4af-home" (OuterVolumeSpecName: "home") pod "145b5179-374e-498b-a7ef-1a70201eb4af" (UID: "145b5179-374e-498b-a7ef-1a70201eb4af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:27.827970 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.827831    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mx2c7\" (UniqueName: \"kubernetes.io/projected/145b5179-374e-498b-a7ef-1a70201eb4af-kube-api-access-mx2c7\") pod \"145b5179-374e-498b-a7ef-1a70201eb4af\" (UID: \"145b5179-374e-498b-a7ef-1a70201eb4af\") "
Sep 11 18:35:27.827970 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.827874    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/145b5179-374e-498b-a7ef-1a70201eb4af-dshm\") pod \"145b5179-374e-498b-a7ef-1a70201eb4af\" (UID: \"145b5179-374e-498b-a7ef-1a70201eb4af\") "
Sep 11 18:35:27.827970 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.827915    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c73362b7-f9eb-4789-8d84-062491a7dd45-tmp-dir\") pod \"c73362b7-f9eb-4789-8d84-062491a7dd45\" (UID: \"c73362b7-f9eb-4789-8d84-062491a7dd45\") "
Sep 11 18:35:27.827970 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.827949    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c73362b7-f9eb-4789-8d84-062491a7dd45-home\") pod \"c73362b7-f9eb-4789-8d84-062491a7dd45\" (UID: \"c73362b7-f9eb-4789-8d84-062491a7dd45\") "
Sep 11 18:35:27.828108 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.827991    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/145b5179-374e-498b-a7ef-1a70201eb4af-tmp-dir\") pod \"145b5179-374e-498b-a7ef-1a70201eb4af\" (UID: \"145b5179-374e-498b-a7ef-1a70201eb4af\") "
Sep 11 18:35:27.828108 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.828064    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c73362b7-f9eb-4789-8d84-062491a7dd45-tls-certs\") pod \"c73362b7-f9eb-4789-8d84-062491a7dd45\" (UID: \"c73362b7-f9eb-4789-8d84-062491a7dd45\") "
Sep 11 18:35:27.828185 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.828120    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k56lv\" (UniqueName: \"kubernetes.io/projected/c73362b7-f9eb-4789-8d84-062491a7dd45-kube-api-access-k56lv\") pod \"c73362b7-f9eb-4789-8d84-062491a7dd45\" (UID: \"c73362b7-f9eb-4789-8d84-062491a7dd45\") "
Sep 11 18:35:27.828893 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.828502    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/145b5179-374e-498b-a7ef-1a70201eb4af-home\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:35:27.828893 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.828527    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/145b5179-374e-498b-a7ef-1a70201eb4af-model-cache\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:35:27.828893 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.828541    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c73362b7-f9eb-4789-8d84-062491a7dd45-model-cache\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:35:27.828893 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.828639    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73362b7-f9eb-4789-8d84-062491a7dd45-home" (OuterVolumeSpecName: "home") pod "c73362b7-f9eb-4789-8d84-062491a7dd45" (UID: "c73362b7-f9eb-4789-8d84-062491a7dd45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:27.829335 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.828924    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/145b5179-374e-498b-a7ef-1a70201eb4af-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "145b5179-374e-498b-a7ef-1a70201eb4af" (UID: "145b5179-374e-498b-a7ef-1a70201eb4af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:27.829335 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.829205    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73362b7-f9eb-4789-8d84-062491a7dd45-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c73362b7-f9eb-4789-8d84-062491a7dd45" (UID: "c73362b7-f9eb-4789-8d84-062491a7dd45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:27.830320 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.830288    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73362b7-f9eb-4789-8d84-062491a7dd45-dshm" (OuterVolumeSpecName: "dshm") pod "c73362b7-f9eb-4789-8d84-062491a7dd45" (UID: "c73362b7-f9eb-4789-8d84-062491a7dd45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:27.830434 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.830340    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/145b5179-374e-498b-a7ef-1a70201eb4af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "145b5179-374e-498b-a7ef-1a70201eb4af" (UID: "145b5179-374e-498b-a7ef-1a70201eb4af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:35:27.830505 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.830436    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145b5179-374e-498b-a7ef-1a70201eb4af-kube-api-access-mx2c7" (OuterVolumeSpecName: "kube-api-access-mx2c7") pod "145b5179-374e-498b-a7ef-1a70201eb4af" (UID: "145b5179-374e-498b-a7ef-1a70201eb4af"). InnerVolumeSpecName "kube-api-access-mx2c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:35:27.831174 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.831153    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c73362b7-f9eb-4789-8d84-062491a7dd45-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c73362b7-f9eb-4789-8d84-062491a7dd45" (UID: "c73362b7-f9eb-4789-8d84-062491a7dd45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:35:27.831484 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.831455    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/145b5179-374e-498b-a7ef-1a70201eb4af-dshm" (OuterVolumeSpecName: "dshm") pod "145b5179-374e-498b-a7ef-1a70201eb4af" (UID: "145b5179-374e-498b-a7ef-1a70201eb4af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:27.831827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.831809    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73362b7-f9eb-4789-8d84-062491a7dd45-kube-api-access-k56lv" (OuterVolumeSpecName: "kube-api-access-k56lv") pod "c73362b7-f9eb-4789-8d84-062491a7dd45" (UID: "c73362b7-f9eb-4789-8d84-062491a7dd45"). InnerVolumeSpecName "kube-api-access-k56lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:35:27.844943 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.844913    2576 generic.go:356] "Generic (PLEG): container finished" podID="c73362b7-f9eb-4789-8d84-062491a7dd45" containerID="d8e424845dea51271bf5f59eb74799855bc3e7483118b21dc876a414b27b6e10" exitCode=0
Sep 11 18:35:27.845061 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.844957    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"
Sep 11 18:35:27.845061 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.844998    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n" event={"ID":"c73362b7-f9eb-4789-8d84-062491a7dd45","Type":"ContainerDied","Data":"d8e424845dea51271bf5f59eb74799855bc3e7483118b21dc876a414b27b6e10"}
Sep 11 18:35:27.845061 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.845040    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n" event={"ID":"c73362b7-f9eb-4789-8d84-062491a7dd45","Type":"ContainerDied","Data":"d6942ac8d2c98383dc7593792534198536cb11dd1e076d8bfbc9fda08b917198"}
Sep 11 18:35:27.845222 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.845063    2576 scope.go:117] "RemoveContainer" containerID="d8e424845dea51271bf5f59eb74799855bc3e7483118b21dc876a414b27b6e10"
Sep 11 18:35:27.846683 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.846575    2576 generic.go:356] "Generic (PLEG): container finished" podID="145b5179-374e-498b-a7ef-1a70201eb4af" containerID="c4d9ffbf3da9990e890e3b60092b42bfbe1dd90a14afe7cfcbfa4a83f68a1068" exitCode=0
Sep 11 18:35:27.846683 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.846651    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"
Sep 11 18:35:27.846843 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.846681    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f" event={"ID":"145b5179-374e-498b-a7ef-1a70201eb4af","Type":"ContainerDied","Data":"c4d9ffbf3da9990e890e3b60092b42bfbe1dd90a14afe7cfcbfa4a83f68a1068"}
Sep 11 18:35:27.846843 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.846718    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f" event={"ID":"145b5179-374e-498b-a7ef-1a70201eb4af","Type":"ContainerDied","Data":"b7865b57ada579e49a550d8fa8991faa73cdfc591a941dc56da10d9b091f6a6b"}
Sep 11 18:35:27.854958 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.854938    2576 scope.go:117] "RemoveContainer" containerID="d8e424845dea51271bf5f59eb74799855bc3e7483118b21dc876a414b27b6e10"
Sep 11 18:35:27.855297 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:35:27.855272    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8e424845dea51271bf5f59eb74799855bc3e7483118b21dc876a414b27b6e10\": container with ID starting with d8e424845dea51271bf5f59eb74799855bc3e7483118b21dc876a414b27b6e10 not found: ID does not exist" containerID="d8e424845dea51271bf5f59eb74799855bc3e7483118b21dc876a414b27b6e10"
Sep 11 18:35:27.855389 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.855305    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8e424845dea51271bf5f59eb74799855bc3e7483118b21dc876a414b27b6e10"} err="failed to get container status \"d8e424845dea51271bf5f59eb74799855bc3e7483118b21dc876a414b27b6e10\": rpc error: code = NotFound desc = could not find container \"d8e424845dea51271bf5f59eb74799855bc3e7483118b21dc876a414b27b6e10\": container with ID starting with d8e424845dea51271bf5f59eb74799855bc3e7483118b21dc876a414b27b6e10 not found: ID does not exist"
Sep 11 18:35:27.855389 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.855328    2576 scope.go:117] "RemoveContainer" containerID="c4d9ffbf3da9990e890e3b60092b42bfbe1dd90a14afe7cfcbfa4a83f68a1068"
Sep 11 18:35:27.864739 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.864723    2576 scope.go:117] "RemoveContainer" containerID="c4d9ffbf3da9990e890e3b60092b42bfbe1dd90a14afe7cfcbfa4a83f68a1068"
Sep 11 18:35:27.864980 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:35:27.864964    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d9ffbf3da9990e890e3b60092b42bfbe1dd90a14afe7cfcbfa4a83f68a1068\": container with ID starting with c4d9ffbf3da9990e890e3b60092b42bfbe1dd90a14afe7cfcbfa4a83f68a1068 not found: ID does not exist" containerID="c4d9ffbf3da9990e890e3b60092b42bfbe1dd90a14afe7cfcbfa4a83f68a1068"
Sep 11 18:35:27.865046 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.864985    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d9ffbf3da9990e890e3b60092b42bfbe1dd90a14afe7cfcbfa4a83f68a1068"} err="failed to get container status \"c4d9ffbf3da9990e890e3b60092b42bfbe1dd90a14afe7cfcbfa4a83f68a1068\": rpc error: code = NotFound desc = could not find container \"c4d9ffbf3da9990e890e3b60092b42bfbe1dd90a14afe7cfcbfa4a83f68a1068\": container with ID starting with c4d9ffbf3da9990e890e3b60092b42bfbe1dd90a14afe7cfcbfa4a83f68a1068 not found: ID does not exist"
Sep 11 18:35:27.869183 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.869162    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"]
Sep 11 18:35:27.874611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.874575    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d73031/distinct-w0-d73031-kserve-68ff5f68fc-4z24n"]
Sep 11 18:35:27.885496 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.885476    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"]
Sep 11 18:35:27.888908 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.888886    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d73031/embedding-w0-d73031-kserve-5764dbf97d-5k56f"]
Sep 11 18:35:27.911025 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.911000    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="145b5179-374e-498b-a7ef-1a70201eb4af" path="/var/lib/kubelet/pods/145b5179-374e-498b-a7ef-1a70201eb4af/volumes"
Sep 11 18:35:27.911404 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.911391    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c73362b7-f9eb-4789-8d84-062491a7dd45" path="/var/lib/kubelet/pods/c73362b7-f9eb-4789-8d84-062491a7dd45/volumes"
Sep 11 18:35:27.929283 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.929260    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k56lv\" (UniqueName: \"kubernetes.io/projected/c73362b7-f9eb-4789-8d84-062491a7dd45-kube-api-access-k56lv\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:35:27.929368 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.929286    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/145b5179-374e-498b-a7ef-1a70201eb4af-tls-certs\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:35:27.929368 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.929295    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c73362b7-f9eb-4789-8d84-062491a7dd45-dshm\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:35:27.929368 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.929306    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx2c7\" (UniqueName: \"kubernetes.io/projected/145b5179-374e-498b-a7ef-1a70201eb4af-kube-api-access-mx2c7\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:35:27.929368 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.929314    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/145b5179-374e-498b-a7ef-1a70201eb4af-dshm\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:35:27.929368 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.929322    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c73362b7-f9eb-4789-8d84-062491a7dd45-tmp-dir\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:35:27.929368 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.929331    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c73362b7-f9eb-4789-8d84-062491a7dd45-home\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:35:27.929368 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.929338    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/145b5179-374e-498b-a7ef-1a70201eb4af-tmp-dir\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:35:27.929368 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:27.929345    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c73362b7-f9eb-4789-8d84-062491a7dd45-tls-certs\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:35:54.410400 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:35:54.410365    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:36:07.186707 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:07.186671    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e370a-9h4lz"]
Sep 11 18:36:07.187086 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:07.187035    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="145b5179-374e-498b-a7ef-1a70201eb4af" containerName="main"
Sep 11 18:36:07.187086 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:07.187046    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="145b5179-374e-498b-a7ef-1a70201eb4af" containerName="main"
Sep 11 18:36:07.187086 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:07.187062    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c73362b7-f9eb-4789-8d84-062491a7dd45" containerName="main"
Sep 11 18:36:07.187086 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:07.187068    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c73362b7-f9eb-4789-8d84-062491a7dd45" containerName="main"
Sep 11 18:36:07.187086 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:07.187085    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13435514-2ef4-401b-b2d6-cfe8fd0f52ef" containerName="payload-pre-processing"
Sep 11 18:36:07.187253 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:07.187090    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="13435514-2ef4-401b-b2d6-cfe8fd0f52ef" containerName="payload-pre-processing"
Sep 11 18:36:07.187253 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:07.187144    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="145b5179-374e-498b-a7ef-1a70201eb4af" containerName="main"
Sep 11 18:36:07.187253 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:07.187154    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="13435514-2ef4-401b-b2d6-cfe8fd0f52ef" containerName="payload-pre-processing"
Sep 11 18:36:07.187253 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:07.187161    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c73362b7-f9eb-4789-8d84-062491a7dd45" containerName="main"
Sep 11 18:36:07.190176 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:07.190153    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e370a-9h4lz"
Sep 11 18:36:07.192346 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:07.192322    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t9kdx\""
Sep 11 18:36:07.197518 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:07.197486    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e370a-9h4lz"]
Sep 11 18:36:07.287129 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:07.287081    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8407c1d4-692e-4a2a-a998-bb4bef9f546a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1e370a-9h4lz\" (UID: \"8407c1d4-692e-4a2a-a998-bb4bef9f546a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e370a-9h4lz"
Sep 11 18:36:07.388025 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:07.387977    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8407c1d4-692e-4a2a-a998-bb4bef9f546a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1e370a-9h4lz\" (UID: \"8407c1d4-692e-4a2a-a998-bb4bef9f546a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e370a-9h4lz"
Sep 11 18:36:07.388667 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:07.388647    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8407c1d4-692e-4a2a-a998-bb4bef9f546a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1e370a-9h4lz\" (UID: \"8407c1d4-692e-4a2a-a998-bb4bef9f546a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e370a-9h4lz"
Sep 11 18:36:07.502575 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:07.502533    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e370a-9h4lz"
Sep 11 18:36:07.781711 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:07.781521    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e370a-9h4lz"]
Sep 11 18:36:07.784599 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:36:07.784557    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8407c1d4_692e_4a2a_a998_bb4bef9f546a.slice/crio-772645bc32f4b81f3c71b011f9017d974571c8e9377f643851b5ede61800c48a WatchSource:0}: Error finding container 772645bc32f4b81f3c71b011f9017d974571c8e9377f643851b5ede61800c48a: Status 404 returned error can't find the container with id 772645bc32f4b81f3c71b011f9017d974571c8e9377f643851b5ede61800c48a
Sep 11 18:36:08.058890 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:08.058789    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e370a-9h4lz" event={"ID":"8407c1d4-692e-4a2a-a998-bb4bef9f546a","Type":"ContainerStarted","Data":"f94e02198587ebc65d3ab21e98facc24cedb68718ae6b38b6883a61497a97cb8"}
Sep 11 18:36:08.058890 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:08.058851    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e370a-9h4lz" event={"ID":"8407c1d4-692e-4a2a-a998-bb4bef9f546a","Type":"ContainerStarted","Data":"772645bc32f4b81f3c71b011f9017d974571c8e9377f643851b5ede61800c48a"}
Sep 11 18:36:08.073210 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:08.073156    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e370a-9h4lz" podStartSLOduration=1.073133806 podStartE2EDuration="1.073133806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:36:08.072452768 +0000 UTC m=+1566.673813932" watchObservedRunningTime="2026-09-11 18:36:08.073133806 +0000 UTC m=+1566.674495013"
Sep 11 18:36:09.065409 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:09.065375    2576 generic.go:356] "Generic (PLEG): container finished" podID="8407c1d4-692e-4a2a-a998-bb4bef9f546a" containerID="f94e02198587ebc65d3ab21e98facc24cedb68718ae6b38b6883a61497a97cb8" exitCode=0
Sep 11 18:36:09.065840 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:09.065457    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e370a-9h4lz" event={"ID":"8407c1d4-692e-4a2a-a998-bb4bef9f546a","Type":"ContainerDied","Data":"f94e02198587ebc65d3ab21e98facc24cedb68718ae6b38b6883a61497a97cb8"}
Sep 11 18:36:10.208714 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:10.208682    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e370a-9h4lz"
Sep 11 18:36:10.316127 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:10.316088    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8407c1d4-692e-4a2a-a998-bb4bef9f546a-maas-api-service-ca\") pod \"8407c1d4-692e-4a2a-a998-bb4bef9f546a\" (UID: \"8407c1d4-692e-4a2a-a998-bb4bef9f546a\") "
Sep 11 18:36:10.316453 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:10.316431    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8407c1d4-692e-4a2a-a998-bb4bef9f546a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8407c1d4-692e-4a2a-a998-bb4bef9f546a" (UID: "8407c1d4-692e-4a2a-a998-bb4bef9f546a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:36:10.417497 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:10.417400    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8407c1d4-692e-4a2a-a998-bb4bef9f546a-maas-api-service-ca\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:11.076964 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:11.076925    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e370a-9h4lz" event={"ID":"8407c1d4-692e-4a2a-a998-bb4bef9f546a","Type":"ContainerDied","Data":"772645bc32f4b81f3c71b011f9017d974571c8e9377f643851b5ede61800c48a"}
Sep 11 18:36:11.077169 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:11.076982    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="772645bc32f4b81f3c71b011f9017d974571c8e9377f643851b5ede61800c48a"
Sep 11 18:36:11.077169 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:11.076938    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e370a-9h4lz"
Sep 11 18:36:19.818294 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:19.818257    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:36:23.165551 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:23.165514    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t"]
Sep 11 18:36:23.166017 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:23.165789    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t" podUID="70a4b88d-51fa-4dc4-8359-65ac102c172a" containerName="payload-processing" containerID="cri-o://56d2b91f83fbfe5044cebc2056037ec2787e79c6299cc216c709741c13f03dc9" gracePeriod=30
Sep 11 18:36:23.343693 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:23.343662    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e370a-9h4lz"]
Sep 11 18:36:23.348155 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:23.348125    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e370a-9h4lz"]
Sep 11 18:36:23.421910 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:23.421851    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t"
Sep 11 18:36:23.540137 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:23.540104    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85m7v\" (UniqueName: \"kubernetes.io/projected/70a4b88d-51fa-4dc4-8359-65ac102c172a-kube-api-access-85m7v\") pod \"70a4b88d-51fa-4dc4-8359-65ac102c172a\" (UID: \"70a4b88d-51fa-4dc4-8359-65ac102c172a\") "
Sep 11 18:36:23.540339 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:23.540174    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a4b88d-51fa-4dc4-8359-65ac102c172a-plugins-config-volume\") pod \"70a4b88d-51fa-4dc4-8359-65ac102c172a\" (UID: \"70a4b88d-51fa-4dc4-8359-65ac102c172a\") "
Sep 11 18:36:23.540600 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:23.540554    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a4b88d-51fa-4dc4-8359-65ac102c172a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70a4b88d-51fa-4dc4-8359-65ac102c172a" (UID: "70a4b88d-51fa-4dc4-8359-65ac102c172a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:36:23.542124 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:23.542105    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a4b88d-51fa-4dc4-8359-65ac102c172a-kube-api-access-85m7v" (OuterVolumeSpecName: "kube-api-access-85m7v") pod "70a4b88d-51fa-4dc4-8359-65ac102c172a" (UID: "70a4b88d-51fa-4dc4-8359-65ac102c172a"). InnerVolumeSpecName "kube-api-access-85m7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:36:23.641854 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:23.641817    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85m7v\" (UniqueName: \"kubernetes.io/projected/70a4b88d-51fa-4dc4-8359-65ac102c172a-kube-api-access-85m7v\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:23.641854 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:23.641849    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a4b88d-51fa-4dc4-8359-65ac102c172a-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:23.910317 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:23.910286    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8407c1d4-692e-4a2a-a998-bb4bef9f546a" path="/var/lib/kubelet/pods/8407c1d4-692e-4a2a-a998-bb4bef9f546a/volumes"
Sep 11 18:36:24.143170 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:24.143138    2576 generic.go:356] "Generic (PLEG): container finished" podID="70a4b88d-51fa-4dc4-8359-65ac102c172a" containerID="56d2b91f83fbfe5044cebc2056037ec2787e79c6299cc216c709741c13f03dc9" exitCode=0
Sep 11 18:36:24.143336 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:24.143201    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t"
Sep 11 18:36:24.143336 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:24.143226    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t" event={"ID":"70a4b88d-51fa-4dc4-8359-65ac102c172a","Type":"ContainerDied","Data":"56d2b91f83fbfe5044cebc2056037ec2787e79c6299cc216c709741c13f03dc9"}
Sep 11 18:36:24.143336 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:24.143287    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t" event={"ID":"70a4b88d-51fa-4dc4-8359-65ac102c172a","Type":"ContainerDied","Data":"63bcf42639d34017ae9853ede71dd7e052fe8f2ce4ab0c59eddfe267950eb2b5"}
Sep 11 18:36:24.143336 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:24.143309    2576 scope.go:117] "RemoveContainer" containerID="56d2b91f83fbfe5044cebc2056037ec2787e79c6299cc216c709741c13f03dc9"
Sep 11 18:36:24.152063 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:24.152045    2576 scope.go:117] "RemoveContainer" containerID="56d2b91f83fbfe5044cebc2056037ec2787e79c6299cc216c709741c13f03dc9"
Sep 11 18:36:24.152387 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:36:24.152355    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56d2b91f83fbfe5044cebc2056037ec2787e79c6299cc216c709741c13f03dc9\": container with ID starting with 56d2b91f83fbfe5044cebc2056037ec2787e79c6299cc216c709741c13f03dc9 not found: ID does not exist" containerID="56d2b91f83fbfe5044cebc2056037ec2787e79c6299cc216c709741c13f03dc9"
Sep 11 18:36:24.152534 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:24.152394    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56d2b91f83fbfe5044cebc2056037ec2787e79c6299cc216c709741c13f03dc9"} err="failed to get container status \"56d2b91f83fbfe5044cebc2056037ec2787e79c6299cc216c709741c13f03dc9\": rpc error: code = NotFound desc = could not find container \"56d2b91f83fbfe5044cebc2056037ec2787e79c6299cc216c709741c13f03dc9\": container with ID starting with 56d2b91f83fbfe5044cebc2056037ec2787e79c6299cc216c709741c13f03dc9 not found: ID does not exist"
Sep 11 18:36:24.160898 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:24.160841    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t"]
Sep 11 18:36:24.164032 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:24.164010    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1e370a-5f88dfcd7c-f8d8t"]
Sep 11 18:36:25.910617 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:25.910566    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70a4b88d-51fa-4dc4-8359-65ac102c172a" path="/var/lib/kubelet/pods/70a4b88d-51fa-4dc4-8359-65ac102c172a/volumes"
Sep 11 18:36:30.847772 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:30.847738    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b77bc8-jmr4c"]
Sep 11 18:36:30.848137 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:30.848085    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70a4b88d-51fa-4dc4-8359-65ac102c172a" containerName="payload-processing"
Sep 11 18:36:30.848137 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:30.848097    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="70a4b88d-51fa-4dc4-8359-65ac102c172a" containerName="payload-processing"
Sep 11 18:36:30.848137 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:30.848111    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8407c1d4-692e-4a2a-a998-bb4bef9f546a" containerName="revoke-keys"
Sep 11 18:36:30.848137 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:30.848116    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8407c1d4-692e-4a2a-a998-bb4bef9f546a" containerName="revoke-keys"
Sep 11 18:36:30.848278 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:30.848183    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8407c1d4-692e-4a2a-a998-bb4bef9f546a" containerName="revoke-keys"
Sep 11 18:36:30.848278 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:30.848193    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="70a4b88d-51fa-4dc4-8359-65ac102c172a" containerName="payload-processing"
Sep 11 18:36:30.852812 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:30.852796    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b77bc8-jmr4c"
Sep 11 18:36:30.855280 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:30.855260    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t9kdx\""
Sep 11 18:36:30.858713 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:30.858690    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b77bc8-jmr4c"]
Sep 11 18:36:31.010635 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:31.010571    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98dfa06-0fa8-41ad-8dd9-ac9c60922493-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b77bc8-jmr4c\" (UID: \"d98dfa06-0fa8-41ad-8dd9-ac9c60922493\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b77bc8-jmr4c"
Sep 11 18:36:31.111168 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:31.111065    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98dfa06-0fa8-41ad-8dd9-ac9c60922493-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b77bc8-jmr4c\" (UID: \"d98dfa06-0fa8-41ad-8dd9-ac9c60922493\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b77bc8-jmr4c"
Sep 11 18:36:31.111764 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:31.111740    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98dfa06-0fa8-41ad-8dd9-ac9c60922493-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b77bc8-jmr4c\" (UID: \"d98dfa06-0fa8-41ad-8dd9-ac9c60922493\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b77bc8-jmr4c"
Sep 11 18:36:31.165735 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:31.165701    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b77bc8-jmr4c"
Sep 11 18:36:31.435123 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:31.435098    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b77bc8-jmr4c"]
Sep 11 18:36:31.437364 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:36:31.437339    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98dfa06_0fa8_41ad_8dd9_ac9c60922493.slice/crio-a78962549f8d1c5ea5b0cc39f54eadbef588ad6f56f8e710bf54df834d643220 WatchSource:0}: Error finding container a78962549f8d1c5ea5b0cc39f54eadbef588ad6f56f8e710bf54df834d643220: Status 404 returned error can't find the container with id a78962549f8d1c5ea5b0cc39f54eadbef588ad6f56f8e710bf54df834d643220
Sep 11 18:36:32.185509 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:32.185419    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b77bc8-jmr4c" event={"ID":"d98dfa06-0fa8-41ad-8dd9-ac9c60922493","Type":"ContainerStarted","Data":"2aefcba4ca52ec1f8536ca564405e69151f81f29c819ac5de34719cd520e3a28"}
Sep 11 18:36:32.185509 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:32.185480    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b77bc8-jmr4c" event={"ID":"d98dfa06-0fa8-41ad-8dd9-ac9c60922493","Type":"ContainerStarted","Data":"a78962549f8d1c5ea5b0cc39f54eadbef588ad6f56f8e710bf54df834d643220"}
Sep 11 18:36:32.200270 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:32.200215    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b77bc8-jmr4c" podStartSLOduration=2.200196436 podStartE2EDuration="2.200196436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:36:32.199864332 +0000 UTC m=+1590.801225499" watchObservedRunningTime="2026-09-11 18:36:32.200196436 +0000 UTC m=+1590.801557599"
Sep 11 18:36:33.192424 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:33.192390    2576 generic.go:356] "Generic (PLEG): container finished" podID="d98dfa06-0fa8-41ad-8dd9-ac9c60922493" containerID="2aefcba4ca52ec1f8536ca564405e69151f81f29c819ac5de34719cd520e3a28" exitCode=0
Sep 11 18:36:33.192832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:33.192486    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b77bc8-jmr4c" event={"ID":"d98dfa06-0fa8-41ad-8dd9-ac9c60922493","Type":"ContainerDied","Data":"2aefcba4ca52ec1f8536ca564405e69151f81f29c819ac5de34719cd520e3a28"}
Sep 11 18:36:34.326306 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:34.326279    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b77bc8-jmr4c"
Sep 11 18:36:34.441608 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:34.441554    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98dfa06-0fa8-41ad-8dd9-ac9c60922493-maas-api-service-ca\") pod \"d98dfa06-0fa8-41ad-8dd9-ac9c60922493\" (UID: \"d98dfa06-0fa8-41ad-8dd9-ac9c60922493\") "
Sep 11 18:36:34.441944 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:34.441922    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d98dfa06-0fa8-41ad-8dd9-ac9c60922493-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d98dfa06-0fa8-41ad-8dd9-ac9c60922493" (UID: "d98dfa06-0fa8-41ad-8dd9-ac9c60922493"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:36:34.542710 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:34.542624    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98dfa06-0fa8-41ad-8dd9-ac9c60922493-maas-api-service-ca\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:35.204402 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:35.204372    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b77bc8-jmr4c" event={"ID":"d98dfa06-0fa8-41ad-8dd9-ac9c60922493","Type":"ContainerDied","Data":"a78962549f8d1c5ea5b0cc39f54eadbef588ad6f56f8e710bf54df834d643220"}
Sep 11 18:36:35.204402 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:35.204391    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b77bc8-jmr4c"
Sep 11 18:36:35.204626 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:35.204419    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a78962549f8d1c5ea5b0cc39f54eadbef588ad6f56f8e710bf54df834d643220"
Sep 11 18:36:37.444725 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:37.444685    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-553596-6hs6h"]
Sep 11 18:36:37.445116 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:37.445068    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d98dfa06-0fa8-41ad-8dd9-ac9c60922493" containerName="revoke-keys"
Sep 11 18:36:37.445116 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:37.445080    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d98dfa06-0fa8-41ad-8dd9-ac9c60922493" containerName="revoke-keys"
Sep 11 18:36:37.445188 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:37.445155    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d98dfa06-0fa8-41ad-8dd9-ac9c60922493" containerName="revoke-keys"
Sep 11 18:36:37.448072 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:37.448050    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-553596-6hs6h"
Sep 11 18:36:37.450872 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:37.450846    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t9kdx\""
Sep 11 18:36:37.456511 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:37.456476    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-553596-6hs6h"]
Sep 11 18:36:37.572028 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:37.571978    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92dc9764-cfed-421f-8c2c-fd38f3e71426-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-553596-6hs6h\" (UID: \"92dc9764-cfed-421f-8c2c-fd38f3e71426\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-553596-6hs6h"
Sep 11 18:36:37.673278 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:37.673240    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92dc9764-cfed-421f-8c2c-fd38f3e71426-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-553596-6hs6h\" (UID: \"92dc9764-cfed-421f-8c2c-fd38f3e71426\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-553596-6hs6h"
Sep 11 18:36:37.673974 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:37.673947    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92dc9764-cfed-421f-8c2c-fd38f3e71426-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-553596-6hs6h\" (UID: \"92dc9764-cfed-421f-8c2c-fd38f3e71426\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-553596-6hs6h"
Sep 11 18:36:37.759614 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:37.759489    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-553596-6hs6h"
Sep 11 18:36:38.036046 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:38.035960    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-553596-6hs6h"]
Sep 11 18:36:38.039780 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:36:38.039751    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92dc9764_cfed_421f_8c2c_fd38f3e71426.slice/crio-5510cf12a961b847dabdb7ac315b0baac5c90d71f4f2107362258d0d527a521c WatchSource:0}: Error finding container 5510cf12a961b847dabdb7ac315b0baac5c90d71f4f2107362258d0d527a521c: Status 404 returned error can't find the container with id 5510cf12a961b847dabdb7ac315b0baac5c90d71f4f2107362258d0d527a521c
Sep 11 18:36:38.221316 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:38.221283    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-553596-6hs6h" event={"ID":"92dc9764-cfed-421f-8c2c-fd38f3e71426","Type":"ContainerStarted","Data":"5510cf12a961b847dabdb7ac315b0baac5c90d71f4f2107362258d0d527a521c"}
Sep 11 18:36:39.228665 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:39.228629    2576 generic.go:356] "Generic (PLEG): container finished" podID="92dc9764-cfed-421f-8c2c-fd38f3e71426" containerID="ba1e71cc9a01664c90b765066e2333dfd9612c1b0101d77aefc59177ff9549a3" exitCode=0
Sep 11 18:36:39.229125 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:39.228716    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-553596-6hs6h" event={"ID":"92dc9764-cfed-421f-8c2c-fd38f3e71426","Type":"ContainerDied","Data":"ba1e71cc9a01664c90b765066e2333dfd9612c1b0101d77aefc59177ff9549a3"}
Sep 11 18:36:40.373566 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:40.373534    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-553596-6hs6h"
Sep 11 18:36:40.503429 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:40.503385    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92dc9764-cfed-421f-8c2c-fd38f3e71426-maas-api-service-ca\") pod \"92dc9764-cfed-421f-8c2c-fd38f3e71426\" (UID: \"92dc9764-cfed-421f-8c2c-fd38f3e71426\") "
Sep 11 18:36:40.503800 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:40.503776    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92dc9764-cfed-421f-8c2c-fd38f3e71426-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "92dc9764-cfed-421f-8c2c-fd38f3e71426" (UID: "92dc9764-cfed-421f-8c2c-fd38f3e71426"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:36:40.604521 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:40.604480    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92dc9764-cfed-421f-8c2c-fd38f3e71426-maas-api-service-ca\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:41.240209 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:41.240179    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-553596-6hs6h"
Sep 11 18:36:41.240209 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:41.240199    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-553596-6hs6h" event={"ID":"92dc9764-cfed-421f-8c2c-fd38f3e71426","Type":"ContainerDied","Data":"5510cf12a961b847dabdb7ac315b0baac5c90d71f4f2107362258d0d527a521c"}
Sep 11 18:36:41.240425 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:41.240245    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5510cf12a961b847dabdb7ac315b0baac5c90d71f4f2107362258d0d527a521c"
Sep 11 18:36:42.345599 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:42.345558    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"]
Sep 11 18:36:42.359642 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:42.359606    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"]
Sep 11 18:36:42.376461 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:42.375752    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"]
Sep 11 18:36:47.107668 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:47.107615    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:36:48.829608 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:48.826824    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b77bc8-jmr4c"]
Sep 11 18:36:48.833195 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:48.833157    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b77bc8-jmr4c"]
Sep 11 18:36:49.910628 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:49.910573    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d98dfa06-0fa8-41ad-8dd9-ac9c60922493" path="/var/lib/kubelet/pods/d98dfa06-0fa8-41ad-8dd9-ac9c60922493/volumes"
Sep 11 18:36:53.142708 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:53.142668    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m"]
Sep 11 18:36:53.143143 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:53.142918    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m" podUID="ca9cd8a2-5d2a-4b6e-a016-f771ca54da22" containerName="maas-api" containerID="cri-o://ea34069dd3b25641ab249952ef3dd402be094ae14630e09b359ab9ec2ddc7bc5" gracePeriod=30
Sep 11 18:36:53.305671 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:53.305640    2576 generic.go:356] "Generic (PLEG): container finished" podID="ca9cd8a2-5d2a-4b6e-a016-f771ca54da22" containerID="ea34069dd3b25641ab249952ef3dd402be094ae14630e09b359ab9ec2ddc7bc5" exitCode=0
Sep 11 18:36:53.305857 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:53.305687    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m" event={"ID":"ca9cd8a2-5d2a-4b6e-a016-f771ca54da22","Type":"ContainerDied","Data":"ea34069dd3b25641ab249952ef3dd402be094ae14630e09b359ab9ec2ddc7bc5"}
Sep 11 18:36:53.430598 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:53.430562    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m"
Sep 11 18:36:53.514886 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:53.514799    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-553596-6hs6h"]
Sep 11 18:36:53.522100 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:53.522062    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-553596-6hs6h"]
Sep 11 18:36:53.531225 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:53.531184    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22-metrics-certs\") pod \"ca9cd8a2-5d2a-4b6e-a016-f771ca54da22\" (UID: \"ca9cd8a2-5d2a-4b6e-a016-f771ca54da22\") "
Sep 11 18:36:53.531423 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:53.531379    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22-openshift-service-ca\") pod \"ca9cd8a2-5d2a-4b6e-a016-f771ca54da22\" (UID: \"ca9cd8a2-5d2a-4b6e-a016-f771ca54da22\") "
Sep 11 18:36:53.531480 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:53.531446    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7p6h9\" (UniqueName: \"kubernetes.io/projected/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22-kube-api-access-7p6h9\") pod \"ca9cd8a2-5d2a-4b6e-a016-f771ca54da22\" (UID: \"ca9cd8a2-5d2a-4b6e-a016-f771ca54da22\") "
Sep 11 18:36:53.531923 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:53.531759    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22-maas-api-tls\") pod \"ca9cd8a2-5d2a-4b6e-a016-f771ca54da22\" (UID: \"ca9cd8a2-5d2a-4b6e-a016-f771ca54da22\") "
Sep 11 18:36:53.532166 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:53.532130    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ca9cd8a2-5d2a-4b6e-a016-f771ca54da22" (UID: "ca9cd8a2-5d2a-4b6e-a016-f771ca54da22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:36:53.534299 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:53.534267    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ca9cd8a2-5d2a-4b6e-a016-f771ca54da22" (UID: "ca9cd8a2-5d2a-4b6e-a016-f771ca54da22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:36:53.534502 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:53.534476    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ca9cd8a2-5d2a-4b6e-a016-f771ca54da22" (UID: "ca9cd8a2-5d2a-4b6e-a016-f771ca54da22"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:36:53.534502 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:53.534492    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22-kube-api-access-7p6h9" (OuterVolumeSpecName: "kube-api-access-7p6h9") pod "ca9cd8a2-5d2a-4b6e-a016-f771ca54da22" (UID: "ca9cd8a2-5d2a-4b6e-a016-f771ca54da22"). InnerVolumeSpecName "kube-api-access-7p6h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:36:53.633301 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:53.633252    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p6h9\" (UniqueName: \"kubernetes.io/projected/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22-kube-api-access-7p6h9\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:53.633301 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:53.633291    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22-maas-api-tls\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:53.633301 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:53.633301    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22-metrics-certs\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:53.633301 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:53.633310    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22-openshift-service-ca\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:53.910956 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:53.910915    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92dc9764-cfed-421f-8c2c-fd38f3e71426" path="/var/lib/kubelet/pods/92dc9764-cfed-421f-8c2c-fd38f3e71426/volumes"
Sep 11 18:36:54.311772 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:54.311740    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m"
Sep 11 18:36:54.312188 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:54.311741    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m" event={"ID":"ca9cd8a2-5d2a-4b6e-a016-f771ca54da22","Type":"ContainerDied","Data":"fe21d357d7a5428ef98a63d6e82560df5f2a281084a6cc96fbb03467134c8154"}
Sep 11 18:36:54.312188 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:54.311877    2576 scope.go:117] "RemoveContainer" containerID="ea34069dd3b25641ab249952ef3dd402be094ae14630e09b359ab9ec2ddc7bc5"
Sep 11 18:36:54.330213 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:54.330182    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m"]
Sep 11 18:36:54.336559 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:54.336528    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-553596-68848b9fb5-q4m2m"]
Sep 11 18:36:55.598663 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:55.598626    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"]
Sep 11 18:36:55.913530 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:55.913445    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca9cd8a2-5d2a-4b6e-a016-f771ca54da22" path="/var/lib/kubelet/pods/ca9cd8a2-5d2a-4b6e-a016-f771ca54da22/volumes"
Sep 11 18:36:57.366424 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.366376    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64" podUID="ec02549b-7498-46de-86f6-9b7ea8b2c846" containerName="main" containerID="cri-o://82766962126faad71f1baf3408f9236ffe4d0d4c2af23d424162330d1767241b" gracePeriod=45
Sep 11 18:36:57.383709 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.383677    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr" podUID="d912ceb5-bbf0-44d3-8ea4-0b29099464cc" containerName="main" containerID="cri-o://0fdff7c16d3d26a2572564cf3718497b7ffb716573c964b74bf2d1ee5aad629d" gracePeriod=45
Sep 11 18:36:57.396182 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.396134    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq" podUID="2f798c6d-ed14-4a83-b321-5329ae4d7286" containerName="main" containerID="cri-o://966d292d8eb1d942aa04e160e0a9cca87658a6e59a721e67b6eab498755d6264" gracePeriod=45
Sep 11 18:36:57.551308 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.551276    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a761e9-g9x25"]
Sep 11 18:36:57.551693 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.551667    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92dc9764-cfed-421f-8c2c-fd38f3e71426" containerName="revoke-keys"
Sep 11 18:36:57.551693 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.551686    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="92dc9764-cfed-421f-8c2c-fd38f3e71426" containerName="revoke-keys"
Sep 11 18:36:57.551901 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.551705    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca9cd8a2-5d2a-4b6e-a016-f771ca54da22" containerName="maas-api"
Sep 11 18:36:57.551901 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.551713    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca9cd8a2-5d2a-4b6e-a016-f771ca54da22" containerName="maas-api"
Sep 11 18:36:57.551901 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.551803    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="92dc9764-cfed-421f-8c2c-fd38f3e71426" containerName="revoke-keys"
Sep 11 18:36:57.551901 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.551819    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca9cd8a2-5d2a-4b6e-a016-f771ca54da22" containerName="maas-api"
Sep 11 18:36:57.554908 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.554888    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a761e9-g9x25"
Sep 11 18:36:57.557415 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.557392    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t9kdx\""
Sep 11 18:36:57.557597 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.557571    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:36:57.568710 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.568614    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a761e9-g9x25"]
Sep 11 18:36:57.663860 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.663830    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:36:57.670197 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.670156    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a7665e-1957-401c-a32f-2ee7c0f7c680-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a761e9-g9x25\" (UID: \"e7a7665e-1957-401c-a32f-2ee7c0f7c680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a761e9-g9x25"
Sep 11 18:36:57.706929 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.706897    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:36:57.710187 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.710160    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:36:57.771222 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.771184    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec02549b-7498-46de-86f6-9b7ea8b2c846-dshm\") pod \"ec02549b-7498-46de-86f6-9b7ea8b2c846\" (UID: \"ec02549b-7498-46de-86f6-9b7ea8b2c846\") "
Sep 11 18:36:57.771397 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.771230    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f798c6d-ed14-4a83-b321-5329ae4d7286-model-cache\") pod \"2f798c6d-ed14-4a83-b321-5329ae4d7286\" (UID: \"2f798c6d-ed14-4a83-b321-5329ae4d7286\") "
Sep 11 18:36:57.771397 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.771254    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-model-cache\") pod \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\" (UID: \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\") "
Sep 11 18:36:57.771397 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.771304    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cwf8\" (UniqueName: \"kubernetes.io/projected/ec02549b-7498-46de-86f6-9b7ea8b2c846-kube-api-access-4cwf8\") pod \"ec02549b-7498-46de-86f6-9b7ea8b2c846\" (UID: \"ec02549b-7498-46de-86f6-9b7ea8b2c846\") "
Sep 11 18:36:57.771397 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.771329    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec02549b-7498-46de-86f6-9b7ea8b2c846-home\") pod \"ec02549b-7498-46de-86f6-9b7ea8b2c846\" (UID: \"ec02549b-7498-46de-86f6-9b7ea8b2c846\") "
Sep 11 18:36:57.771397 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.771355    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwx7s\" (UniqueName: \"kubernetes.io/projected/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-kube-api-access-fwx7s\") pod \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\" (UID: \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\") "
Sep 11 18:36:57.771397 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.771382    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f798c6d-ed14-4a83-b321-5329ae4d7286-tmp-dir\") pod \"2f798c6d-ed14-4a83-b321-5329ae4d7286\" (UID: \"2f798c6d-ed14-4a83-b321-5329ae4d7286\") "
Sep 11 18:36:57.771767 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.771407    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f798c6d-ed14-4a83-b321-5329ae4d7286-home\") pod \"2f798c6d-ed14-4a83-b321-5329ae4d7286\" (UID: \"2f798c6d-ed14-4a83-b321-5329ae4d7286\") "
Sep 11 18:36:57.771767 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.771429    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-tmp-dir\") pod \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\" (UID: \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\") "
Sep 11 18:36:57.771767 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.771482    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec02549b-7498-46de-86f6-9b7ea8b2c846-tls-certs\") pod \"ec02549b-7498-46de-86f6-9b7ea8b2c846\" (UID: \"ec02549b-7498-46de-86f6-9b7ea8b2c846\") "
Sep 11 18:36:57.771767 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.771518    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-dshm\") pod \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\" (UID: \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\") "
Sep 11 18:36:57.771767 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.771594    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-tls-certs\") pod \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\" (UID: \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\") "
Sep 11 18:36:57.771767 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.771642    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jknbx\" (UniqueName: \"kubernetes.io/projected/2f798c6d-ed14-4a83-b321-5329ae4d7286-kube-api-access-jknbx\") pod \"2f798c6d-ed14-4a83-b321-5329ae4d7286\" (UID: \"2f798c6d-ed14-4a83-b321-5329ae4d7286\") "
Sep 11 18:36:57.771767 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.771676    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec02549b-7498-46de-86f6-9b7ea8b2c846-model-cache\") pod \"ec02549b-7498-46de-86f6-9b7ea8b2c846\" (UID: \"ec02549b-7498-46de-86f6-9b7ea8b2c846\") "
Sep 11 18:36:57.771767 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.771514    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f798c6d-ed14-4a83-b321-5329ae4d7286-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f798c6d-ed14-4a83-b321-5329ae4d7286" (UID: "2f798c6d-ed14-4a83-b321-5329ae4d7286"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:36:57.771767 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.771740    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec02549b-7498-46de-86f6-9b7ea8b2c846-tmp-dir\") pod \"ec02549b-7498-46de-86f6-9b7ea8b2c846\" (UID: \"ec02549b-7498-46de-86f6-9b7ea8b2c846\") "
Sep 11 18:36:57.771767 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.771768    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-home\") pod \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\" (UID: \"d912ceb5-bbf0-44d3-8ea4-0b29099464cc\") "
Sep 11 18:36:57.772301 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.771791    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f798c6d-ed14-4a83-b321-5329ae4d7286-tls-certs\") pod \"2f798c6d-ed14-4a83-b321-5329ae4d7286\" (UID: \"2f798c6d-ed14-4a83-b321-5329ae4d7286\") "
Sep 11 18:36:57.772301 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.771814    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f798c6d-ed14-4a83-b321-5329ae4d7286-dshm\") pod \"2f798c6d-ed14-4a83-b321-5329ae4d7286\" (UID: \"2f798c6d-ed14-4a83-b321-5329ae4d7286\") "
Sep 11 18:36:57.772301 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.771991    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a7665e-1957-401c-a32f-2ee7c0f7c680-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a761e9-g9x25\" (UID: \"e7a7665e-1957-401c-a32f-2ee7c0f7c680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a761e9-g9x25"
Sep 11 18:36:57.772301 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.772147    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f798c6d-ed14-4a83-b321-5329ae4d7286-model-cache\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:57.774777 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.774088    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a7665e-1957-401c-a32f-2ee7c0f7c680-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a761e9-g9x25\" (UID: \"e7a7665e-1957-401c-a32f-2ee7c0f7c680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a761e9-g9x25"
Sep 11 18:36:57.775624 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.771542    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-model-cache" (OuterVolumeSpecName: "model-cache") pod "d912ceb5-bbf0-44d3-8ea4-0b29099464cc" (UID: "d912ceb5-bbf0-44d3-8ea4-0b29099464cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:36:57.775624 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.771717    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f798c6d-ed14-4a83-b321-5329ae4d7286-home" (OuterVolumeSpecName: "home") pod "2f798c6d-ed14-4a83-b321-5329ae4d7286" (UID: "2f798c6d-ed14-4a83-b321-5329ae4d7286"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:36:57.775624 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.772023    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f798c6d-ed14-4a83-b321-5329ae4d7286-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f798c6d-ed14-4a83-b321-5329ae4d7286" (UID: "2f798c6d-ed14-4a83-b321-5329ae4d7286"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:36:57.775624 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.772269    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec02549b-7498-46de-86f6-9b7ea8b2c846-home" (OuterVolumeSpecName: "home") pod "ec02549b-7498-46de-86f6-9b7ea8b2c846" (UID: "ec02549b-7498-46de-86f6-9b7ea8b2c846"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:36:57.775624 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.772412    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec02549b-7498-46de-86f6-9b7ea8b2c846-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ec02549b-7498-46de-86f6-9b7ea8b2c846" (UID: "ec02549b-7498-46de-86f6-9b7ea8b2c846"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:36:57.775624 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.773932    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d912ceb5-bbf0-44d3-8ea4-0b29099464cc" (UID: "d912ceb5-bbf0-44d3-8ea4-0b29099464cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:36:57.775624 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.774318    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-home" (OuterVolumeSpecName: "home") pod "d912ceb5-bbf0-44d3-8ea4-0b29099464cc" (UID: "d912ceb5-bbf0-44d3-8ea4-0b29099464cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:36:57.775624 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.774623    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec02549b-7498-46de-86f6-9b7ea8b2c846-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec02549b-7498-46de-86f6-9b7ea8b2c846" (UID: "ec02549b-7498-46de-86f6-9b7ea8b2c846"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:36:57.775624 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.774688    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec02549b-7498-46de-86f6-9b7ea8b2c846-kube-api-access-4cwf8" (OuterVolumeSpecName: "kube-api-access-4cwf8") pod "ec02549b-7498-46de-86f6-9b7ea8b2c846" (UID: "ec02549b-7498-46de-86f6-9b7ea8b2c846"). InnerVolumeSpecName "kube-api-access-4cwf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:36:57.775624 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.774746    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec02549b-7498-46de-86f6-9b7ea8b2c846-dshm" (OuterVolumeSpecName: "dshm") pod "ec02549b-7498-46de-86f6-9b7ea8b2c846" (UID: "ec02549b-7498-46de-86f6-9b7ea8b2c846"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:36:57.775624 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.775010    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec02549b-7498-46de-86f6-9b7ea8b2c846-model-cache" (OuterVolumeSpecName: "model-cache") pod "ec02549b-7498-46de-86f6-9b7ea8b2c846" (UID: "ec02549b-7498-46de-86f6-9b7ea8b2c846"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:36:57.784571 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.784530    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-kube-api-access-fwx7s" (OuterVolumeSpecName: "kube-api-access-fwx7s") pod "d912ceb5-bbf0-44d3-8ea4-0b29099464cc" (UID: "d912ceb5-bbf0-44d3-8ea4-0b29099464cc"). InnerVolumeSpecName "kube-api-access-fwx7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:36:57.784571 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.784562    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f798c6d-ed14-4a83-b321-5329ae4d7286-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f798c6d-ed14-4a83-b321-5329ae4d7286" (UID: "2f798c6d-ed14-4a83-b321-5329ae4d7286"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:36:57.784813 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.784563    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f798c6d-ed14-4a83-b321-5329ae4d7286-dshm" (OuterVolumeSpecName: "dshm") pod "2f798c6d-ed14-4a83-b321-5329ae4d7286" (UID: "2f798c6d-ed14-4a83-b321-5329ae4d7286"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:36:57.784813 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.784545    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-dshm" (OuterVolumeSpecName: "dshm") pod "d912ceb5-bbf0-44d3-8ea4-0b29099464cc" (UID: "d912ceb5-bbf0-44d3-8ea4-0b29099464cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:36:57.784813 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.784545    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d912ceb5-bbf0-44d3-8ea4-0b29099464cc" (UID: "d912ceb5-bbf0-44d3-8ea4-0b29099464cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:36:57.786523 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.786498    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f798c6d-ed14-4a83-b321-5329ae4d7286-kube-api-access-jknbx" (OuterVolumeSpecName: "kube-api-access-jknbx") pod "2f798c6d-ed14-4a83-b321-5329ae4d7286" (UID: "2f798c6d-ed14-4a83-b321-5329ae4d7286"). InnerVolumeSpecName "kube-api-access-jknbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:36:57.871430 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.871388    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a761e9-g9x25"
Sep 11 18:36:57.872825 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.872801    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec02549b-7498-46de-86f6-9b7ea8b2c846-tls-certs\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:57.872825 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.872826    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-dshm\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:57.872956 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.872835    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-tls-certs\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:57.872956 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.872846    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jknbx\" (UniqueName: \"kubernetes.io/projected/2f798c6d-ed14-4a83-b321-5329ae4d7286-kube-api-access-jknbx\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:57.872956 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.872856    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec02549b-7498-46de-86f6-9b7ea8b2c846-model-cache\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:57.872956 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.872865    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec02549b-7498-46de-86f6-9b7ea8b2c846-tmp-dir\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:57.872956 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.872871    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-home\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:57.872956 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.872879    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f798c6d-ed14-4a83-b321-5329ae4d7286-tls-certs\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:57.872956 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.872886    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f798c6d-ed14-4a83-b321-5329ae4d7286-dshm\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:57.872956 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.872893    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec02549b-7498-46de-86f6-9b7ea8b2c846-dshm\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:57.872956 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.872899    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-model-cache\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:57.872956 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.872907    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cwf8\" (UniqueName: \"kubernetes.io/projected/ec02549b-7498-46de-86f6-9b7ea8b2c846-kube-api-access-4cwf8\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:57.872956 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.872914    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec02549b-7498-46de-86f6-9b7ea8b2c846-home\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:57.872956 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.872923    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwx7s\" (UniqueName: \"kubernetes.io/projected/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-kube-api-access-fwx7s\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:57.872956 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.872932    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f798c6d-ed14-4a83-b321-5329ae4d7286-tmp-dir\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:57.872956 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.872939    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f798c6d-ed14-4a83-b321-5329ae4d7286-home\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:57.872956 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:57.872946    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d912ceb5-bbf0-44d3-8ea4-0b29099464cc-tmp-dir\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:36:58.145918 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.145892    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a761e9-g9x25"]
Sep 11 18:36:58.148184 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:36:58.148155    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a7665e_1957_401c_a32f_2ee7c0f7c680.slice/crio-753aec7531d9e4332589cd05e5849864c1c4afb60e648e15200fe594be7df23e WatchSource:0}: Error finding container 753aec7531d9e4332589cd05e5849864c1c4afb60e648e15200fe594be7df23e: Status 404 returned error can't find the container with id 753aec7531d9e4332589cd05e5849864c1c4afb60e648e15200fe594be7df23e
Sep 11 18:36:58.335567 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.335526    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a761e9-g9x25" event={"ID":"e7a7665e-1957-401c-a32f-2ee7c0f7c680","Type":"ContainerStarted","Data":"753aec7531d9e4332589cd05e5849864c1c4afb60e648e15200fe594be7df23e"}
Sep 11 18:36:58.336871 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.336844    2576 generic.go:356] "Generic (PLEG): container finished" podID="d912ceb5-bbf0-44d3-8ea4-0b29099464cc" containerID="0fdff7c16d3d26a2572564cf3718497b7ffb716573c964b74bf2d1ee5aad629d" exitCode=0
Sep 11 18:36:58.337024 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.336912    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"
Sep 11 18:36:58.337024 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.336961    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr" event={"ID":"d912ceb5-bbf0-44d3-8ea4-0b29099464cc","Type":"ContainerDied","Data":"0fdff7c16d3d26a2572564cf3718497b7ffb716573c964b74bf2d1ee5aad629d"}
Sep 11 18:36:58.337144 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.337049    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr" event={"ID":"d912ceb5-bbf0-44d3-8ea4-0b29099464cc","Type":"ContainerDied","Data":"00e2529f2ae2d3d19362545dc736a21856fd72e033975c334045db05beff1bab"}
Sep 11 18:36:58.337144 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.337081    2576 scope.go:117] "RemoveContainer" containerID="0fdff7c16d3d26a2572564cf3718497b7ffb716573c964b74bf2d1ee5aad629d"
Sep 11 18:36:58.338522 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.338488    2576 generic.go:356] "Generic (PLEG): container finished" podID="2f798c6d-ed14-4a83-b321-5329ae4d7286" containerID="966d292d8eb1d942aa04e160e0a9cca87658a6e59a721e67b6eab498755d6264" exitCode=0
Sep 11 18:36:58.338667 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.338542    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"
Sep 11 18:36:58.338667 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.338567    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq" event={"ID":"2f798c6d-ed14-4a83-b321-5329ae4d7286","Type":"ContainerDied","Data":"966d292d8eb1d942aa04e160e0a9cca87658a6e59a721e67b6eab498755d6264"}
Sep 11 18:36:58.338667 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.338622    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq" event={"ID":"2f798c6d-ed14-4a83-b321-5329ae4d7286","Type":"ContainerDied","Data":"7faf8bb34381491abb0f9a19089ef1549a0cc46b450964a5069b89524037aca9"}
Sep 11 18:36:58.340175 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.340143    2576 generic.go:356] "Generic (PLEG): container finished" podID="ec02549b-7498-46de-86f6-9b7ea8b2c846" containerID="82766962126faad71f1baf3408f9236ffe4d0d4c2af23d424162330d1767241b" exitCode=0
Sep 11 18:36:58.340273 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.340213    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"
Sep 11 18:36:58.340273 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.340222    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64" event={"ID":"ec02549b-7498-46de-86f6-9b7ea8b2c846","Type":"ContainerDied","Data":"82766962126faad71f1baf3408f9236ffe4d0d4c2af23d424162330d1767241b"}
Sep 11 18:36:58.340391 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.340273    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64" event={"ID":"ec02549b-7498-46de-86f6-9b7ea8b2c846","Type":"ContainerDied","Data":"6575ec84716bcdb2a6f00f2583b427ddb3f77916b914f5d60aaa96e59d480270"}
Sep 11 18:36:58.347313 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.347290    2576 scope.go:117] "RemoveContainer" containerID="0fdff7c16d3d26a2572564cf3718497b7ffb716573c964b74bf2d1ee5aad629d"
Sep 11 18:36:58.347691 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:36:58.347668    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fdff7c16d3d26a2572564cf3718497b7ffb716573c964b74bf2d1ee5aad629d\": container with ID starting with 0fdff7c16d3d26a2572564cf3718497b7ffb716573c964b74bf2d1ee5aad629d not found: ID does not exist" containerID="0fdff7c16d3d26a2572564cf3718497b7ffb716573c964b74bf2d1ee5aad629d"
Sep 11 18:36:58.347796 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.347697    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fdff7c16d3d26a2572564cf3718497b7ffb716573c964b74bf2d1ee5aad629d"} err="failed to get container status \"0fdff7c16d3d26a2572564cf3718497b7ffb716573c964b74bf2d1ee5aad629d\": rpc error: code = NotFound desc = could not find container \"0fdff7c16d3d26a2572564cf3718497b7ffb716573c964b74bf2d1ee5aad629d\": container with ID starting with 0fdff7c16d3d26a2572564cf3718497b7ffb716573c964b74bf2d1ee5aad629d not found: ID does not exist"
Sep 11 18:36:58.347796 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.347716    2576 scope.go:117] "RemoveContainer" containerID="966d292d8eb1d942aa04e160e0a9cca87658a6e59a721e67b6eab498755d6264"
Sep 11 18:36:58.359455 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.359434    2576 scope.go:117] "RemoveContainer" containerID="966d292d8eb1d942aa04e160e0a9cca87658a6e59a721e67b6eab498755d6264"
Sep 11 18:36:58.359831 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:36:58.359810    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"966d292d8eb1d942aa04e160e0a9cca87658a6e59a721e67b6eab498755d6264\": container with ID starting with 966d292d8eb1d942aa04e160e0a9cca87658a6e59a721e67b6eab498755d6264 not found: ID does not exist" containerID="966d292d8eb1d942aa04e160e0a9cca87658a6e59a721e67b6eab498755d6264"
Sep 11 18:36:58.359912 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.359841    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"966d292d8eb1d942aa04e160e0a9cca87658a6e59a721e67b6eab498755d6264"} err="failed to get container status \"966d292d8eb1d942aa04e160e0a9cca87658a6e59a721e67b6eab498755d6264\": rpc error: code = NotFound desc = could not find container \"966d292d8eb1d942aa04e160e0a9cca87658a6e59a721e67b6eab498755d6264\": container with ID starting with 966d292d8eb1d942aa04e160e0a9cca87658a6e59a721e67b6eab498755d6264 not found: ID does not exist"
Sep 11 18:36:58.359912 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.359865    2576 scope.go:117] "RemoveContainer" containerID="82766962126faad71f1baf3408f9236ffe4d0d4c2af23d424162330d1767241b"
Sep 11 18:36:58.368721 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.368692    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"]
Sep 11 18:36:58.369018 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.369010    2576 scope.go:117] "RemoveContainer" containerID="82766962126faad71f1baf3408f9236ffe4d0d4c2af23d424162330d1767241b"
Sep 11 18:36:58.369349 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:36:58.369328    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82766962126faad71f1baf3408f9236ffe4d0d4c2af23d424162330d1767241b\": container with ID starting with 82766962126faad71f1baf3408f9236ffe4d0d4c2af23d424162330d1767241b not found: ID does not exist" containerID="82766962126faad71f1baf3408f9236ffe4d0d4c2af23d424162330d1767241b"
Sep 11 18:36:58.369393 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.369358    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82766962126faad71f1baf3408f9236ffe4d0d4c2af23d424162330d1767241b"} err="failed to get container status \"82766962126faad71f1baf3408f9236ffe4d0d4c2af23d424162330d1767241b\": rpc error: code = NotFound desc = could not find container \"82766962126faad71f1baf3408f9236ffe4d0d4c2af23d424162330d1767241b\": container with ID starting with 82766962126faad71f1baf3408f9236ffe4d0d4c2af23d424162330d1767241b not found: ID does not exist"
Sep 11 18:36:58.372607 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.372573    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-553596/simulator-w4-553596-kserve-7c74f776f4-4z7zq"]
Sep 11 18:36:58.385553 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.385525    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"]
Sep 11 18:36:58.389171 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.389145    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-553596/premium-w4-553596-kserve-65c5f4f79-97nzr"]
Sep 11 18:36:58.397938 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.397869    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"]
Sep 11 18:36:58.404034 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.404002    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-553596/distinct-2-w4-553596-kserve-cc4bb45b9-2kz64"]
Sep 11 18:36:58.945421 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:58.945385    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"]
Sep 11 18:36:59.347890 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:59.347855    2576 generic.go:356] "Generic (PLEG): container finished" podID="e7a7665e-1957-401c-a32f-2ee7c0f7c680" containerID="ed7cba147b3e89af4008b18d1ad3c6297144a18d37f87b364a535409d6dd684a" exitCode=0
Sep 11 18:36:59.348071 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:59.347934    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a761e9-g9x25" event={"ID":"e7a7665e-1957-401c-a32f-2ee7c0f7c680","Type":"ContainerDied","Data":"ed7cba147b3e89af4008b18d1ad3c6297144a18d37f87b364a535409d6dd684a"}
Sep 11 18:36:59.910845 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:59.910808    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f798c6d-ed14-4a83-b321-5329ae4d7286" path="/var/lib/kubelet/pods/2f798c6d-ed14-4a83-b321-5329ae4d7286/volumes"
Sep 11 18:36:59.911222 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:59.911209    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d912ceb5-bbf0-44d3-8ea4-0b29099464cc" path="/var/lib/kubelet/pods/d912ceb5-bbf0-44d3-8ea4-0b29099464cc/volumes"
Sep 11 18:36:59.911555 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:36:59.911542    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec02549b-7498-46de-86f6-9b7ea8b2c846" path="/var/lib/kubelet/pods/ec02549b-7498-46de-86f6-9b7ea8b2c846/volumes"
Sep 11 18:37:00.495207 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:00.495179    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a761e9-g9x25"
Sep 11 18:37:00.599389 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:00.599348    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a7665e-1957-401c-a32f-2ee7c0f7c680-maas-api-service-ca\") pod \"e7a7665e-1957-401c-a32f-2ee7c0f7c680\" (UID: \"e7a7665e-1957-401c-a32f-2ee7c0f7c680\") "
Sep 11 18:37:00.599756 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:00.599733    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7a7665e-1957-401c-a32f-2ee7c0f7c680-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e7a7665e-1957-401c-a32f-2ee7c0f7c680" (UID: "e7a7665e-1957-401c-a32f-2ee7c0f7c680"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:37:00.700248 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:00.700217    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a7665e-1957-401c-a32f-2ee7c0f7c680-maas-api-service-ca\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:37:01.363468 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:01.363432    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a761e9-g9x25" event={"ID":"e7a7665e-1957-401c-a32f-2ee7c0f7c680","Type":"ContainerDied","Data":"753aec7531d9e4332589cd05e5849864c1c4afb60e648e15200fe594be7df23e"}
Sep 11 18:37:01.363897 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:01.363481    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="753aec7531d9e4332589cd05e5849864c1c4afb60e648e15200fe594be7df23e"
Sep 11 18:37:01.363897 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:01.363444    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a761e9-g9x25"
Sep 11 18:37:08.357126 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:08.357086    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt"]
Sep 11 18:37:08.365062 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:08.365027    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb"]
Sep 11 18:37:08.365271 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:08.365246    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb" podUID="6546fdce-36a2-4a07-a06c-4786da61d190" containerName="payload-pre-processing" containerID="cri-o://1def032bfe962225706e52bc8fa8a078b5c72da5bda5ff6357a26f64322e6365" gracePeriod=30
Sep 11 18:37:08.404813 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:08.404752    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt" podUID="e14a667c-e9e9-41ba-8bd7-105565a09431" containerName="payload-processing" containerID="cri-o://2968876b30887f1771508613e59cb3f8bd85648de170f40a51146952e78be5c5" gracePeriod=30
Sep 11 18:37:08.527822 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:08.527766    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a761e9-g9x25"]
Sep 11 18:37:08.530249 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:08.530217    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a761e9-g9x25"]
Sep 11 18:37:08.663206 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:08.663176    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb"
Sep 11 18:37:08.677812 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:08.677782    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt"
Sep 11 18:37:08.775708 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:08.775666    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14a667c-e9e9-41ba-8bd7-105565a09431-plugins-config-volume\") pod \"e14a667c-e9e9-41ba-8bd7-105565a09431\" (UID: \"e14a667c-e9e9-41ba-8bd7-105565a09431\") "
Sep 11 18:37:08.775708 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:08.775719    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6546fdce-36a2-4a07-a06c-4786da61d190-plugins-config-volume\") pod \"6546fdce-36a2-4a07-a06c-4786da61d190\" (UID: \"6546fdce-36a2-4a07-a06c-4786da61d190\") "
Sep 11 18:37:08.775943 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:08.775752    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zv8v\" (UniqueName: \"kubernetes.io/projected/6546fdce-36a2-4a07-a06c-4786da61d190-kube-api-access-9zv8v\") pod \"6546fdce-36a2-4a07-a06c-4786da61d190\" (UID: \"6546fdce-36a2-4a07-a06c-4786da61d190\") "
Sep 11 18:37:08.775943 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:08.775798    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tl7cj\" (UniqueName: \"kubernetes.io/projected/e14a667c-e9e9-41ba-8bd7-105565a09431-kube-api-access-tl7cj\") pod \"e14a667c-e9e9-41ba-8bd7-105565a09431\" (UID: \"e14a667c-e9e9-41ba-8bd7-105565a09431\") "
Sep 11 18:37:08.776102 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:08.776075    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e14a667c-e9e9-41ba-8bd7-105565a09431-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e14a667c-e9e9-41ba-8bd7-105565a09431" (UID: "e14a667c-e9e9-41ba-8bd7-105565a09431"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:37:08.776179 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:08.776116    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6546fdce-36a2-4a07-a06c-4786da61d190-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6546fdce-36a2-4a07-a06c-4786da61d190" (UID: "6546fdce-36a2-4a07-a06c-4786da61d190"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:37:08.777990 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:08.777965    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14a667c-e9e9-41ba-8bd7-105565a09431-kube-api-access-tl7cj" (OuterVolumeSpecName: "kube-api-access-tl7cj") pod "e14a667c-e9e9-41ba-8bd7-105565a09431" (UID: "e14a667c-e9e9-41ba-8bd7-105565a09431"). InnerVolumeSpecName "kube-api-access-tl7cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:37:08.778082 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:08.778017    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6546fdce-36a2-4a07-a06c-4786da61d190-kube-api-access-9zv8v" (OuterVolumeSpecName: "kube-api-access-9zv8v") pod "6546fdce-36a2-4a07-a06c-4786da61d190" (UID: "6546fdce-36a2-4a07-a06c-4786da61d190"). InnerVolumeSpecName "kube-api-access-9zv8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:37:08.877345 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:08.877298    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6546fdce-36a2-4a07-a06c-4786da61d190-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:37:08.877345 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:08.877332    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zv8v\" (UniqueName: \"kubernetes.io/projected/6546fdce-36a2-4a07-a06c-4786da61d190-kube-api-access-9zv8v\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:37:08.877345 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:08.877344    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl7cj\" (UniqueName: \"kubernetes.io/projected/e14a667c-e9e9-41ba-8bd7-105565a09431-kube-api-access-tl7cj\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:37:08.877345 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:08.877358    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14a667c-e9e9-41ba-8bd7-105565a09431-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:37:09.411732 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:09.411698    2576 generic.go:356] "Generic (PLEG): container finished" podID="6546fdce-36a2-4a07-a06c-4786da61d190" containerID="1def032bfe962225706e52bc8fa8a078b5c72da5bda5ff6357a26f64322e6365" exitCode=0
Sep 11 18:37:09.412165 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:09.411746    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb"
Sep 11 18:37:09.412165 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:09.411786    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb" event={"ID":"6546fdce-36a2-4a07-a06c-4786da61d190","Type":"ContainerDied","Data":"1def032bfe962225706e52bc8fa8a078b5c72da5bda5ff6357a26f64322e6365"}
Sep 11 18:37:09.412165 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:09.411849    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb" event={"ID":"6546fdce-36a2-4a07-a06c-4786da61d190","Type":"ContainerDied","Data":"c00076b6c3b22cb479f162bb46b8ca3121c65ece67eaef620cd82c3e58a61a2e"}
Sep 11 18:37:09.412165 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:09.411876    2576 scope.go:117] "RemoveContainer" containerID="1def032bfe962225706e52bc8fa8a078b5c72da5bda5ff6357a26f64322e6365"
Sep 11 18:37:09.413256 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:09.413218    2576 generic.go:356] "Generic (PLEG): container finished" podID="e14a667c-e9e9-41ba-8bd7-105565a09431" containerID="2968876b30887f1771508613e59cb3f8bd85648de170f40a51146952e78be5c5" exitCode=0
Sep 11 18:37:09.413319 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:09.413281    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt"
Sep 11 18:37:09.413372 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:09.413283    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt" event={"ID":"e14a667c-e9e9-41ba-8bd7-105565a09431","Type":"ContainerDied","Data":"2968876b30887f1771508613e59cb3f8bd85648de170f40a51146952e78be5c5"}
Sep 11 18:37:09.413411 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:09.413398    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt" event={"ID":"e14a667c-e9e9-41ba-8bd7-105565a09431","Type":"ContainerDied","Data":"4265cd45c64a188bfdcef4c5c64819ebb3845265c52a4982daac8960be5f1e43"}
Sep 11 18:37:09.422906 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:09.422883    2576 scope.go:117] "RemoveContainer" containerID="1def032bfe962225706e52bc8fa8a078b5c72da5bda5ff6357a26f64322e6365"
Sep 11 18:37:09.423282 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:37:09.423255    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1def032bfe962225706e52bc8fa8a078b5c72da5bda5ff6357a26f64322e6365\": container with ID starting with 1def032bfe962225706e52bc8fa8a078b5c72da5bda5ff6357a26f64322e6365 not found: ID does not exist" containerID="1def032bfe962225706e52bc8fa8a078b5c72da5bda5ff6357a26f64322e6365"
Sep 11 18:37:09.423386 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:09.423293    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1def032bfe962225706e52bc8fa8a078b5c72da5bda5ff6357a26f64322e6365"} err="failed to get container status \"1def032bfe962225706e52bc8fa8a078b5c72da5bda5ff6357a26f64322e6365\": rpc error: code = NotFound desc = could not find container \"1def032bfe962225706e52bc8fa8a078b5c72da5bda5ff6357a26f64322e6365\": container with ID starting with 1def032bfe962225706e52bc8fa8a078b5c72da5bda5ff6357a26f64322e6365 not found: ID does not exist"
Sep 11 18:37:09.423386 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:09.423320    2576 scope.go:117] "RemoveContainer" containerID="2968876b30887f1771508613e59cb3f8bd85648de170f40a51146952e78be5c5"
Sep 11 18:37:09.431914 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:09.431884    2576 scope.go:117] "RemoveContainer" containerID="2968876b30887f1771508613e59cb3f8bd85648de170f40a51146952e78be5c5"
Sep 11 18:37:09.432385 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:37:09.432361    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2968876b30887f1771508613e59cb3f8bd85648de170f40a51146952e78be5c5\": container with ID starting with 2968876b30887f1771508613e59cb3f8bd85648de170f40a51146952e78be5c5 not found: ID does not exist" containerID="2968876b30887f1771508613e59cb3f8bd85648de170f40a51146952e78be5c5"
Sep 11 18:37:09.432479 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:09.432393    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2968876b30887f1771508613e59cb3f8bd85648de170f40a51146952e78be5c5"} err="failed to get container status \"2968876b30887f1771508613e59cb3f8bd85648de170f40a51146952e78be5c5\": rpc error: code = NotFound desc = could not find container \"2968876b30887f1771508613e59cb3f8bd85648de170f40a51146952e78be5c5\": container with ID starting with 2968876b30887f1771508613e59cb3f8bd85648de170f40a51146952e78be5c5 not found: ID does not exist"
Sep 11 18:37:09.434366 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:09.434340    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb"]
Sep 11 18:37:09.440867 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:09.440838    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a761e9-bbc57c58-ncjsb"]
Sep 11 18:37:09.449689 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:09.449655    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt"]
Sep 11 18:37:09.456000 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:09.455966    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a761e9-8668c4b86c-g5tnt"]
Sep 11 18:37:09.511891 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:09.511857    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:37:09.911366 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:09.911330    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6546fdce-36a2-4a07-a06c-4786da61d190" path="/var/lib/kubelet/pods/6546fdce-36a2-4a07-a06c-4786da61d190/volumes"
Sep 11 18:37:09.911744 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:09.911727    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e14a667c-e9e9-41ba-8bd7-105565a09431" path="/var/lib/kubelet/pods/e14a667c-e9e9-41ba-8bd7-105565a09431/volumes"
Sep 11 18:37:09.912065 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:09.912050    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7a7665e-1957-401c-a32f-2ee7c0f7c680" path="/var/lib/kubelet/pods/e7a7665e-1957-401c-a32f-2ee7c0f7c680/volumes"
Sep 11 18:37:10.617943 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:10.617898    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc" podUID="44b871f0-6d44-437c-9d58-d58f80a85b4f" containerName="main" containerID="cri-o://c5b8672cf5ed7e30a94fbdb337c62cfa7db7878809cc58dcc734c998e22373bf" gracePeriod=45
Sep 11 18:37:10.866635 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:10.866608    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:37:10.999122 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:10.999076    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b871f0-6d44-437c-9d58-d58f80a85b4f-model-cache\") pod \"44b871f0-6d44-437c-9d58-d58f80a85b4f\" (UID: \"44b871f0-6d44-437c-9d58-d58f80a85b4f\") "
Sep 11 18:37:10.999334 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:10.999152    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86nzb\" (UniqueName: \"kubernetes.io/projected/44b871f0-6d44-437c-9d58-d58f80a85b4f-kube-api-access-86nzb\") pod \"44b871f0-6d44-437c-9d58-d58f80a85b4f\" (UID: \"44b871f0-6d44-437c-9d58-d58f80a85b4f\") "
Sep 11 18:37:10.999334 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:10.999216    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b871f0-6d44-437c-9d58-d58f80a85b4f-dshm\") pod \"44b871f0-6d44-437c-9d58-d58f80a85b4f\" (UID: \"44b871f0-6d44-437c-9d58-d58f80a85b4f\") "
Sep 11 18:37:10.999334 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:10.999240    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b871f0-6d44-437c-9d58-d58f80a85b4f-home\") pod \"44b871f0-6d44-437c-9d58-d58f80a85b4f\" (UID: \"44b871f0-6d44-437c-9d58-d58f80a85b4f\") "
Sep 11 18:37:10.999334 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:10.999270    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b871f0-6d44-437c-9d58-d58f80a85b4f-tmp-dir\") pod \"44b871f0-6d44-437c-9d58-d58f80a85b4f\" (UID: \"44b871f0-6d44-437c-9d58-d58f80a85b4f\") "
Sep 11 18:37:10.999334 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:10.999328    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b871f0-6d44-437c-9d58-d58f80a85b4f-tls-certs\") pod \"44b871f0-6d44-437c-9d58-d58f80a85b4f\" (UID: \"44b871f0-6d44-437c-9d58-d58f80a85b4f\") "
Sep 11 18:37:10.999621 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:10.999448    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b871f0-6d44-437c-9d58-d58f80a85b4f-model-cache" (OuterVolumeSpecName: "model-cache") pod "44b871f0-6d44-437c-9d58-d58f80a85b4f" (UID: "44b871f0-6d44-437c-9d58-d58f80a85b4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:10.999621 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:10.999541    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b871f0-6d44-437c-9d58-d58f80a85b4f-home" (OuterVolumeSpecName: "home") pod "44b871f0-6d44-437c-9d58-d58f80a85b4f" (UID: "44b871f0-6d44-437c-9d58-d58f80a85b4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:10.999621 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:10.999557    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b871f0-6d44-437c-9d58-d58f80a85b4f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44b871f0-6d44-437c-9d58-d58f80a85b4f" (UID: "44b871f0-6d44-437c-9d58-d58f80a85b4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:10.999776 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:10.999657    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b871f0-6d44-437c-9d58-d58f80a85b4f-model-cache\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:37:10.999776 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:10.999670    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b871f0-6d44-437c-9d58-d58f80a85b4f-home\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:37:11.001566 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:11.001538    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b871f0-6d44-437c-9d58-d58f80a85b4f-kube-api-access-86nzb" (OuterVolumeSpecName: "kube-api-access-86nzb") pod "44b871f0-6d44-437c-9d58-d58f80a85b4f" (UID: "44b871f0-6d44-437c-9d58-d58f80a85b4f"). InnerVolumeSpecName "kube-api-access-86nzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:37:11.001695 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:11.001596    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b871f0-6d44-437c-9d58-d58f80a85b4f-dshm" (OuterVolumeSpecName: "dshm") pod "44b871f0-6d44-437c-9d58-d58f80a85b4f" (UID: "44b871f0-6d44-437c-9d58-d58f80a85b4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:11.001695 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:11.001613    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44b871f0-6d44-437c-9d58-d58f80a85b4f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44b871f0-6d44-437c-9d58-d58f80a85b4f" (UID: "44b871f0-6d44-437c-9d58-d58f80a85b4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:37:11.100548 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:11.100508    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86nzb\" (UniqueName: \"kubernetes.io/projected/44b871f0-6d44-437c-9d58-d58f80a85b4f-kube-api-access-86nzb\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:37:11.100548 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:11.100541    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b871f0-6d44-437c-9d58-d58f80a85b4f-dshm\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:37:11.100548 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:11.100553    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b871f0-6d44-437c-9d58-d58f80a85b4f-tmp-dir\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:37:11.100548 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:11.100561    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b871f0-6d44-437c-9d58-d58f80a85b4f-tls-certs\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:37:11.426700 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:11.426578    2576 generic.go:356] "Generic (PLEG): container finished" podID="44b871f0-6d44-437c-9d58-d58f80a85b4f" containerID="c5b8672cf5ed7e30a94fbdb337c62cfa7db7878809cc58dcc734c998e22373bf" exitCode=0
Sep 11 18:37:11.426700 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:11.426659    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"
Sep 11 18:37:11.426700 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:11.426688    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc" event={"ID":"44b871f0-6d44-437c-9d58-d58f80a85b4f","Type":"ContainerDied","Data":"c5b8672cf5ed7e30a94fbdb337c62cfa7db7878809cc58dcc734c998e22373bf"}
Sep 11 18:37:11.426917 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:11.426737    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc" event={"ID":"44b871f0-6d44-437c-9d58-d58f80a85b4f","Type":"ContainerDied","Data":"f5387f2bb6d569bf86abac1ff5d08af0a60121b6723491616e39ff5fadf1076c"}
Sep 11 18:37:11.426917 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:11.426761    2576 scope.go:117] "RemoveContainer" containerID="c5b8672cf5ed7e30a94fbdb337c62cfa7db7878809cc58dcc734c998e22373bf"
Sep 11 18:37:11.436314 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:11.436293    2576 scope.go:117] "RemoveContainer" containerID="c5b8672cf5ed7e30a94fbdb337c62cfa7db7878809cc58dcc734c998e22373bf"
Sep 11 18:37:11.436684 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:37:11.436662    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5b8672cf5ed7e30a94fbdb337c62cfa7db7878809cc58dcc734c998e22373bf\": container with ID starting with c5b8672cf5ed7e30a94fbdb337c62cfa7db7878809cc58dcc734c998e22373bf not found: ID does not exist" containerID="c5b8672cf5ed7e30a94fbdb337c62cfa7db7878809cc58dcc734c998e22373bf"
Sep 11 18:37:11.436755 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:11.436699    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b8672cf5ed7e30a94fbdb337c62cfa7db7878809cc58dcc734c998e22373bf"} err="failed to get container status \"c5b8672cf5ed7e30a94fbdb337c62cfa7db7878809cc58dcc734c998e22373bf\": rpc error: code = NotFound desc = could not find container \"c5b8672cf5ed7e30a94fbdb337c62cfa7db7878809cc58dcc734c998e22373bf\": container with ID starting with c5b8672cf5ed7e30a94fbdb337c62cfa7db7878809cc58dcc734c998e22373bf not found: ID does not exist"
Sep 11 18:37:11.447801 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:11.447764    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"]
Sep 11 18:37:11.453503 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:11.453474    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf9898/auth-test-model-bf9898-kserve-65877c4dcf-wdttc"]
Sep 11 18:37:11.911308 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:11.911272    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44b871f0-6d44-437c-9d58-d58f80a85b4f" path="/var/lib/kubelet/pods/44b871f0-6d44-437c-9d58-d58f80a85b4f/volumes"
Sep 11 18:37:13.964603 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:13.964545    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn" podUID="47476a97-dd25-4605-8558-cdc821679b74" containerName="main" containerID="cri-o://d264b2c77e5a6a01cf5f82a273288fb03a225d0e3f4920840c6dbb82a13a5690" gracePeriod=45
Sep 11 18:37:14.223423 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.223356    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:37:14.331643 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.331578    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47476a97-dd25-4605-8558-cdc821679b74-tmp-dir\") pod \"47476a97-dd25-4605-8558-cdc821679b74\" (UID: \"47476a97-dd25-4605-8558-cdc821679b74\") "
Sep 11 18:37:14.331821 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.331658    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wm4k\" (UniqueName: \"kubernetes.io/projected/47476a97-dd25-4605-8558-cdc821679b74-kube-api-access-9wm4k\") pod \"47476a97-dd25-4605-8558-cdc821679b74\" (UID: \"47476a97-dd25-4605-8558-cdc821679b74\") "
Sep 11 18:37:14.331821 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.331699    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47476a97-dd25-4605-8558-cdc821679b74-home\") pod \"47476a97-dd25-4605-8558-cdc821679b74\" (UID: \"47476a97-dd25-4605-8558-cdc821679b74\") "
Sep 11 18:37:14.331821 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.331721    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47476a97-dd25-4605-8558-cdc821679b74-dshm\") pod \"47476a97-dd25-4605-8558-cdc821679b74\" (UID: \"47476a97-dd25-4605-8558-cdc821679b74\") "
Sep 11 18:37:14.331821 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.331790    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47476a97-dd25-4605-8558-cdc821679b74-model-cache\") pod \"47476a97-dd25-4605-8558-cdc821679b74\" (UID: \"47476a97-dd25-4605-8558-cdc821679b74\") "
Sep 11 18:37:14.331975 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.331834    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47476a97-dd25-4605-8558-cdc821679b74-tls-certs\") pod \"47476a97-dd25-4605-8558-cdc821679b74\" (UID: \"47476a97-dd25-4605-8558-cdc821679b74\") "
Sep 11 18:37:14.332021 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.331935    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47476a97-dd25-4605-8558-cdc821679b74-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "47476a97-dd25-4605-8558-cdc821679b74" (UID: "47476a97-dd25-4605-8558-cdc821679b74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:14.332021 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.331995    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47476a97-dd25-4605-8558-cdc821679b74-home" (OuterVolumeSpecName: "home") pod "47476a97-dd25-4605-8558-cdc821679b74" (UID: "47476a97-dd25-4605-8558-cdc821679b74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:14.332106 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.332078    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47476a97-dd25-4605-8558-cdc821679b74-home\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:37:14.332106 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.332090    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47476a97-dd25-4605-8558-cdc821679b74-model-cache" (OuterVolumeSpecName: "model-cache") pod "47476a97-dd25-4605-8558-cdc821679b74" (UID: "47476a97-dd25-4605-8558-cdc821679b74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:14.332213 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.332097    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47476a97-dd25-4605-8558-cdc821679b74-tmp-dir\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:37:14.334101 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.334075    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47476a97-dd25-4605-8558-cdc821679b74-dshm" (OuterVolumeSpecName: "dshm") pod "47476a97-dd25-4605-8558-cdc821679b74" (UID: "47476a97-dd25-4605-8558-cdc821679b74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:14.334101 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.334092    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47476a97-dd25-4605-8558-cdc821679b74-kube-api-access-9wm4k" (OuterVolumeSpecName: "kube-api-access-9wm4k") pod "47476a97-dd25-4605-8558-cdc821679b74" (UID: "47476a97-dd25-4605-8558-cdc821679b74"). InnerVolumeSpecName "kube-api-access-9wm4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:37:14.334101 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.334096    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47476a97-dd25-4605-8558-cdc821679b74-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "47476a97-dd25-4605-8558-cdc821679b74" (UID: "47476a97-dd25-4605-8558-cdc821679b74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:37:14.432558 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.432518    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wm4k\" (UniqueName: \"kubernetes.io/projected/47476a97-dd25-4605-8558-cdc821679b74-kube-api-access-9wm4k\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:37:14.432558 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.432550    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47476a97-dd25-4605-8558-cdc821679b74-dshm\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:37:14.432558 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.432560    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47476a97-dd25-4605-8558-cdc821679b74-model-cache\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:37:14.432558 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.432571    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47476a97-dd25-4605-8558-cdc821679b74-tls-certs\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:37:14.444729 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.444695    2576 generic.go:356] "Generic (PLEG): container finished" podID="47476a97-dd25-4605-8558-cdc821679b74" containerID="d264b2c77e5a6a01cf5f82a273288fb03a225d0e3f4920840c6dbb82a13a5690" exitCode=0
Sep 11 18:37:14.444883 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.444740    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"
Sep 11 18:37:14.444883 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.444778    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn" event={"ID":"47476a97-dd25-4605-8558-cdc821679b74","Type":"ContainerDied","Data":"d264b2c77e5a6a01cf5f82a273288fb03a225d0e3f4920840c6dbb82a13a5690"}
Sep 11 18:37:14.444883 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.444818    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn" event={"ID":"47476a97-dd25-4605-8558-cdc821679b74","Type":"ContainerDied","Data":"4bc51ad0acbd74246dc8197133c99f5f26d6a57ecf092315c8c10e9e399cbaef"}
Sep 11 18:37:14.444883 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.444837    2576 scope.go:117] "RemoveContainer" containerID="d264b2c77e5a6a01cf5f82a273288fb03a225d0e3f4920840c6dbb82a13a5690"
Sep 11 18:37:14.453780 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.453761    2576 scope.go:117] "RemoveContainer" containerID="d264b2c77e5a6a01cf5f82a273288fb03a225d0e3f4920840c6dbb82a13a5690"
Sep 11 18:37:14.454075 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:37:14.454056    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d264b2c77e5a6a01cf5f82a273288fb03a225d0e3f4920840c6dbb82a13a5690\": container with ID starting with d264b2c77e5a6a01cf5f82a273288fb03a225d0e3f4920840c6dbb82a13a5690 not found: ID does not exist" containerID="d264b2c77e5a6a01cf5f82a273288fb03a225d0e3f4920840c6dbb82a13a5690"
Sep 11 18:37:14.454132 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.454087    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d264b2c77e5a6a01cf5f82a273288fb03a225d0e3f4920840c6dbb82a13a5690"} err="failed to get container status \"d264b2c77e5a6a01cf5f82a273288fb03a225d0e3f4920840c6dbb82a13a5690\": rpc error: code = NotFound desc = could not find container \"d264b2c77e5a6a01cf5f82a273288fb03a225d0e3f4920840c6dbb82a13a5690\": container with ID starting with d264b2c77e5a6a01cf5f82a273288fb03a225d0e3f4920840c6dbb82a13a5690 not found: ID does not exist"
Sep 11 18:37:14.464784 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.464750    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"]
Sep 11 18:37:14.468693 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:14.468666    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5faf38/auth-test-model-5faf38-kserve-77bd4b4d5f-9j7hn"]
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.597168    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c3180-mvsws"]
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.597924    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44b871f0-6d44-437c-9d58-d58f80a85b4f" containerName="main"
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.597940    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="44b871f0-6d44-437c-9d58-d58f80a85b4f" containerName="main"
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.597968    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f798c6d-ed14-4a83-b321-5329ae4d7286" containerName="main"
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.597977    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f798c6d-ed14-4a83-b321-5329ae4d7286" containerName="main"
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.597998    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e14a667c-e9e9-41ba-8bd7-105565a09431" containerName="payload-processing"
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.598007    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e14a667c-e9e9-41ba-8bd7-105565a09431" containerName="payload-processing"
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.598017    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7a7665e-1957-401c-a32f-2ee7c0f7c680" containerName="revoke-keys"
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.598032    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7a7665e-1957-401c-a32f-2ee7c0f7c680" containerName="revoke-keys"
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.598045    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6546fdce-36a2-4a07-a06c-4786da61d190" containerName="payload-pre-processing"
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.598053    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6546fdce-36a2-4a07-a06c-4786da61d190" containerName="payload-pre-processing"
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.598063    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47476a97-dd25-4605-8558-cdc821679b74" containerName="main"
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.598070    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="47476a97-dd25-4605-8558-cdc821679b74" containerName="main"
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.598100    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec02549b-7498-46de-86f6-9b7ea8b2c846" containerName="main"
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.598108    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec02549b-7498-46de-86f6-9b7ea8b2c846" containerName="main"
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.598123    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d912ceb5-bbf0-44d3-8ea4-0b29099464cc" containerName="main"
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.598131    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d912ceb5-bbf0-44d3-8ea4-0b29099464cc" containerName="main"
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.598271    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d912ceb5-bbf0-44d3-8ea4-0b29099464cc" containerName="main"
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.598281    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec02549b-7498-46de-86f6-9b7ea8b2c846" containerName="main"
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.598297    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6546fdce-36a2-4a07-a06c-4786da61d190" containerName="payload-pre-processing"
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.598308    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f798c6d-ed14-4a83-b321-5329ae4d7286" containerName="main"
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.598322    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e14a667c-e9e9-41ba-8bd7-105565a09431" containerName="payload-processing"
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.598331    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="44b871f0-6d44-437c-9d58-d58f80a85b4f" containerName="main"
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.598346    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7a7665e-1957-401c-a32f-2ee7c0f7c680" containerName="revoke-keys"
Sep 11 18:37:15.599611 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.598355    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="47476a97-dd25-4605-8558-cdc821679b74" containerName="main"
Sep 11 18:37:15.604744 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.604715    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c3180-mvsws"
Sep 11 18:37:15.607058 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.607023    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:37:15.607058 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.607034    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t9kdx\""
Sep 11 18:37:15.607509 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.607484    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c3180-mvsws"]
Sep 11 18:37:15.743084 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.743038    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/648a70bb-cbcb-4b64-939f-f78ae81b3e63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6c3180-mvsws\" (UID: \"648a70bb-cbcb-4b64-939f-f78ae81b3e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c3180-mvsws"
Sep 11 18:37:15.844579 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.844541    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/648a70bb-cbcb-4b64-939f-f78ae81b3e63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6c3180-mvsws\" (UID: \"648a70bb-cbcb-4b64-939f-f78ae81b3e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c3180-mvsws"
Sep 11 18:37:15.845200 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.845176    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/648a70bb-cbcb-4b64-939f-f78ae81b3e63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6c3180-mvsws\" (UID: \"648a70bb-cbcb-4b64-939f-f78ae81b3e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c3180-mvsws"
Sep 11 18:37:15.910450 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.910368    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47476a97-dd25-4605-8558-cdc821679b74" path="/var/lib/kubelet/pods/47476a97-dd25-4605-8558-cdc821679b74/volumes"
Sep 11 18:37:15.916447 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:15.916413    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c3180-mvsws"
Sep 11 18:37:16.189698 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:16.189670    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c3180-mvsws"]
Sep 11 18:37:16.192096 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:37:16.192069    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648a70bb_cbcb_4b64_939f_f78ae81b3e63.slice/crio-5cc6e3cbb584e1e3b981786c7ed9bc0d94088ac47174be7e73d3cc1c706b3c83 WatchSource:0}: Error finding container 5cc6e3cbb584e1e3b981786c7ed9bc0d94088ac47174be7e73d3cc1c706b3c83: Status 404 returned error can't find the container with id 5cc6e3cbb584e1e3b981786c7ed9bc0d94088ac47174be7e73d3cc1c706b3c83
Sep 11 18:37:16.457248 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:16.457205    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c3180-mvsws" event={"ID":"648a70bb-cbcb-4b64-939f-f78ae81b3e63","Type":"ContainerStarted","Data":"eca9fdadaab02c4880bfca9478c0ea1dd7ecfcfb5a8d1d81b8ebc1dd159c0bb9"}
Sep 11 18:37:16.457431 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:16.457263    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c3180-mvsws" event={"ID":"648a70bb-cbcb-4b64-939f-f78ae81b3e63","Type":"ContainerStarted","Data":"5cc6e3cbb584e1e3b981786c7ed9bc0d94088ac47174be7e73d3cc1c706b3c83"}
Sep 11 18:37:16.474012 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:16.473959    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c3180-mvsws" podStartSLOduration=1.473940387 podStartE2EDuration="1.473940387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:37:16.472892351 +0000 UTC m=+1635.074253516" watchObservedRunningTime="2026-09-11 18:37:16.473940387 +0000 UTC m=+1635.075301549"
Sep 11 18:37:17.464068 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:17.464034    2576 generic.go:356] "Generic (PLEG): container finished" podID="648a70bb-cbcb-4b64-939f-f78ae81b3e63" containerID="eca9fdadaab02c4880bfca9478c0ea1dd7ecfcfb5a8d1d81b8ebc1dd159c0bb9" exitCode=0
Sep 11 18:37:17.464426 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:17.464118    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c3180-mvsws" event={"ID":"648a70bb-cbcb-4b64-939f-f78ae81b3e63","Type":"ContainerDied","Data":"eca9fdadaab02c4880bfca9478c0ea1dd7ecfcfb5a8d1d81b8ebc1dd159c0bb9"}
Sep 11 18:37:18.601223 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:18.601196    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c3180-mvsws"
Sep 11 18:37:18.770464 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:18.770367    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/648a70bb-cbcb-4b64-939f-f78ae81b3e63-maas-api-service-ca\") pod \"648a70bb-cbcb-4b64-939f-f78ae81b3e63\" (UID: \"648a70bb-cbcb-4b64-939f-f78ae81b3e63\") "
Sep 11 18:37:18.770782 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:18.770760    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/648a70bb-cbcb-4b64-939f-f78ae81b3e63-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "648a70bb-cbcb-4b64-939f-f78ae81b3e63" (UID: "648a70bb-cbcb-4b64-939f-f78ae81b3e63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:37:18.777966 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:18.777934    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"]
Sep 11 18:37:18.778262 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:18.778251    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="648a70bb-cbcb-4b64-939f-f78ae81b3e63" containerName="revoke-keys"
Sep 11 18:37:18.778307 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:18.778264    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="648a70bb-cbcb-4b64-939f-f78ae81b3e63" containerName="revoke-keys"
Sep 11 18:37:18.778343 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:18.778328    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="648a70bb-cbcb-4b64-939f-f78ae81b3e63" containerName="revoke-keys"
Sep 11 18:37:18.781128 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:18.781109    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:37:18.783302 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:18.783282    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bf9898\"/\"kube-root-ca.crt\""
Sep 11 18:37:18.783414 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:18.783359    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bf9898\"/\"openshift-service-ca.crt\""
Sep 11 18:37:18.783414 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:18.783282    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bf9898\"/\"default-dockercfg-b87pn\""
Sep 11 18:37:18.783524 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:18.783467    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bf9898\"/\"sub-test-model-bf9898-kserve-self-signed-certs\""
Sep 11 18:37:18.791910 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:18.791887    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"]
Sep 11 18:37:18.871389 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:18.871355    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/648a70bb-cbcb-4b64-939f-f78ae81b3e63-maas-api-service-ca\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:37:18.972212 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:18.972174    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/728286c6-8fa6-44f6-9489-f7df2cd59b0b-model-cache\") pod \"sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg\" (UID: \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:37:18.972212 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:18.972214    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/728286c6-8fa6-44f6-9489-f7df2cd59b0b-tmp-dir\") pod \"sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg\" (UID: \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:37:18.972422 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:18.972266    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/728286c6-8fa6-44f6-9489-f7df2cd59b0b-tls-certs\") pod \"sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg\" (UID: \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:37:18.972422 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:18.972284    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/728286c6-8fa6-44f6-9489-f7df2cd59b0b-dshm\") pod \"sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg\" (UID: \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:37:18.972422 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:18.972370    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/728286c6-8fa6-44f6-9489-f7df2cd59b0b-home\") pod \"sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg\" (UID: \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:37:18.972422 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:18.972406    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9p7r\" (UniqueName: \"kubernetes.io/projected/728286c6-8fa6-44f6-9489-f7df2cd59b0b-kube-api-access-w9p7r\") pod \"sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg\" (UID: \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:37:19.073271 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:19.073174    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/728286c6-8fa6-44f6-9489-f7df2cd59b0b-home\") pod \"sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg\" (UID: \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:37:19.073271 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:19.073214    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9p7r\" (UniqueName: \"kubernetes.io/projected/728286c6-8fa6-44f6-9489-f7df2cd59b0b-kube-api-access-w9p7r\") pod \"sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg\" (UID: \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:37:19.073271 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:19.073259    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/728286c6-8fa6-44f6-9489-f7df2cd59b0b-model-cache\") pod \"sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg\" (UID: \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:37:19.073520 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:19.073374    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/728286c6-8fa6-44f6-9489-f7df2cd59b0b-tmp-dir\") pod \"sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg\" (UID: \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:37:19.073520 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:19.073479    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/728286c6-8fa6-44f6-9489-f7df2cd59b0b-tls-certs\") pod \"sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg\" (UID: \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:37:19.073520 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:19.073515    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/728286c6-8fa6-44f6-9489-f7df2cd59b0b-dshm\") pod \"sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg\" (UID: \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:37:19.073791 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:19.073627    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/728286c6-8fa6-44f6-9489-f7df2cd59b0b-home\") pod \"sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg\" (UID: \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:37:19.073791 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:19.073726    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/728286c6-8fa6-44f6-9489-f7df2cd59b0b-model-cache\") pod \"sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg\" (UID: \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:37:19.073791 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:19.073732    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/728286c6-8fa6-44f6-9489-f7df2cd59b0b-tmp-dir\") pod \"sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg\" (UID: \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:37:19.075756 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:19.075729    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/728286c6-8fa6-44f6-9489-f7df2cd59b0b-dshm\") pod \"sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg\" (UID: \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:37:19.075940 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:19.075924    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/728286c6-8fa6-44f6-9489-f7df2cd59b0b-tls-certs\") pod \"sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg\" (UID: \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:37:19.081690 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:19.081669    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9p7r\" (UniqueName: \"kubernetes.io/projected/728286c6-8fa6-44f6-9489-f7df2cd59b0b-kube-api-access-w9p7r\") pod \"sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg\" (UID: \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:37:19.090568 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:19.090543    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:37:19.372961 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:19.372933    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"]
Sep 11 18:37:19.374958 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:37:19.374928    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod728286c6_8fa6_44f6_9489_f7df2cd59b0b.slice/crio-455708a67defdb176b0211012b340c8f81be495406839de35ecbeba16c3b7199 WatchSource:0}: Error finding container 455708a67defdb176b0211012b340c8f81be495406839de35ecbeba16c3b7199: Status 404 returned error can't find the container with id 455708a67defdb176b0211012b340c8f81be495406839de35ecbeba16c3b7199
Sep 11 18:37:19.481192 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:19.481165    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c3180-mvsws"
Sep 11 18:37:19.481192 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:19.481171    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c3180-mvsws" event={"ID":"648a70bb-cbcb-4b64-939f-f78ae81b3e63","Type":"ContainerDied","Data":"5cc6e3cbb584e1e3b981786c7ed9bc0d94088ac47174be7e73d3cc1c706b3c83"}
Sep 11 18:37:19.481433 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:19.481256    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cc6e3cbb584e1e3b981786c7ed9bc0d94088ac47174be7e73d3cc1c706b3c83"
Sep 11 18:37:19.482989 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:19.482958    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg" event={"ID":"728286c6-8fa6-44f6-9489-f7df2cd59b0b","Type":"ContainerStarted","Data":"16d9947e76be6eaee471c38c370c9266e0507f5f5009687db227b1a54a42f946"}
Sep 11 18:37:19.483120 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:19.482996    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg" event={"ID":"728286c6-8fa6-44f6-9489-f7df2cd59b0b","Type":"ContainerStarted","Data":"455708a67defdb176b0211012b340c8f81be495406839de35ecbeba16c3b7199"}
Sep 11 18:37:19.503913 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:19.503847    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg" podStartSLOduration=1.503826025 podStartE2EDuration="1.503826025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:37:19.501460891 +0000 UTC m=+1638.102822072" watchObservedRunningTime="2026-09-11 18:37:19.503826025 +0000 UTC m=+1638.105187188"
Sep 11 18:37:20.003354 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:20.003312    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:37:26.256022 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:26.255989    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c3180-29819190-gjpzf"]
Sep 11 18:37:26.261420 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:26.261386    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6c3180-29819190-gjpzf"]
Sep 11 18:37:26.291317 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:26.291277    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h"]
Sep 11 18:37:26.291637 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:26.291563    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h" podUID="a1b3a597-b5c6-4da5-914d-4c4ed2781b1e" containerName="payload-processing" containerID="cri-o://a2998737d354526b7ba962252ae74f6fb92f5b2a8f8c1807e9e3e35d8ddf91ac" gracePeriod=30
Sep 11 18:37:26.469272 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:26.469235    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c3180-mvsws"]
Sep 11 18:37:26.470867 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:26.470838    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6c3180-mvsws"]
Sep 11 18:37:26.519492 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:26.519407    2576 generic.go:356] "Generic (PLEG): container finished" podID="a1b3a597-b5c6-4da5-914d-4c4ed2781b1e" containerID="a2998737d354526b7ba962252ae74f6fb92f5b2a8f8c1807e9e3e35d8ddf91ac" exitCode=0
Sep 11 18:37:26.519650 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:26.519483    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h" event={"ID":"a1b3a597-b5c6-4da5-914d-4c4ed2781b1e","Type":"ContainerDied","Data":"a2998737d354526b7ba962252ae74f6fb92f5b2a8f8c1807e9e3e35d8ddf91ac"}
Sep 11 18:37:26.550375 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:26.550338    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h"
Sep 11 18:37:26.629647 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:26.629560    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqrdn\" (UniqueName: \"kubernetes.io/projected/a1b3a597-b5c6-4da5-914d-4c4ed2781b1e-kube-api-access-pqrdn\") pod \"a1b3a597-b5c6-4da5-914d-4c4ed2781b1e\" (UID: \"a1b3a597-b5c6-4da5-914d-4c4ed2781b1e\") "
Sep 11 18:37:26.629812 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:26.629676    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b3a597-b5c6-4da5-914d-4c4ed2781b1e-plugins-config-volume\") pod \"a1b3a597-b5c6-4da5-914d-4c4ed2781b1e\" (UID: \"a1b3a597-b5c6-4da5-914d-4c4ed2781b1e\") "
Sep 11 18:37:26.630058 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:26.630030    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1b3a597-b5c6-4da5-914d-4c4ed2781b1e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a1b3a597-b5c6-4da5-914d-4c4ed2781b1e" (UID: "a1b3a597-b5c6-4da5-914d-4c4ed2781b1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:37:26.631832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:26.631806    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1b3a597-b5c6-4da5-914d-4c4ed2781b1e-kube-api-access-pqrdn" (OuterVolumeSpecName: "kube-api-access-pqrdn") pod "a1b3a597-b5c6-4da5-914d-4c4ed2781b1e" (UID: "a1b3a597-b5c6-4da5-914d-4c4ed2781b1e"). InnerVolumeSpecName "kube-api-access-pqrdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:37:26.730450 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:26.730402    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b3a597-b5c6-4da5-914d-4c4ed2781b1e-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:37:26.730450 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:26.730440    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqrdn\" (UniqueName: \"kubernetes.io/projected/a1b3a597-b5c6-4da5-914d-4c4ed2781b1e-kube-api-access-pqrdn\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:37:27.525998 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:27.525963    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h" event={"ID":"a1b3a597-b5c6-4da5-914d-4c4ed2781b1e","Type":"ContainerDied","Data":"49ac038e7e78f1ce3dfcd717b4e18a14361b89b3d94ded7f9002961725be76a6"}
Sep 11 18:37:27.525998 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:27.525982    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h"
Sep 11 18:37:27.526496 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:27.526033    2576 scope.go:117] "RemoveContainer" containerID="a2998737d354526b7ba962252ae74f6fb92f5b2a8f8c1807e9e3e35d8ddf91ac"
Sep 11 18:37:27.547093 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:27.547056    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h"]
Sep 11 18:37:27.550815 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:27.550781    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6c3180-755b6c74df-m5n9h"]
Sep 11 18:37:27.910052 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:27.909971    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="648a70bb-cbcb-4b64-939f-f78ae81b3e63" path="/var/lib/kubelet/pods/648a70bb-cbcb-4b64-939f-f78ae81b3e63/volumes"
Sep 11 18:37:27.910332 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:27.910319    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1b3a597-b5c6-4da5-914d-4c4ed2781b1e" path="/var/lib/kubelet/pods/a1b3a597-b5c6-4da5-914d-4c4ed2781b1e/volumes"
Sep 11 18:37:27.910561 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:27.910549    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e24483d4-3bd4-4d9d-b40f-305d28b9586a" path="/var/lib/kubelet/pods/e24483d4-3bd4-4d9d-b40f-305d28b9586a/volumes"
Sep 11 18:37:29.091565 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:29.091522    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:37:29.092051 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:29.091603    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:37:29.104237 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:29.104209    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:37:29.303296 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:29.303258    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:37:29.549470 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:37:29.549431    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:38:02.818010 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:02.817972    2576 scope.go:117] "RemoveContainer" containerID="cd9625728d4acb2b1335aa9d0d45d26ad585133eba7f8767684840280418dcf2"
Sep 11 18:38:10.728238 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:10.728200    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:38:27.816343 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:27.816263    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:38:41.402776 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:41.402724    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:38:44.448106 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:44.448070    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7842-791f07"]
Sep 11 18:38:44.448500 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:44.448393    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1b3a597-b5c6-4da5-914d-4c4ed2781b1e" containerName="payload-processing"
Sep 11 18:38:44.448500 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:44.448405    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1b3a597-b5c6-4da5-914d-4c4ed2781b1e" containerName="payload-processing"
Sep 11 18:38:44.448500 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:44.448475    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1b3a597-b5c6-4da5-914d-4c4ed2781b1e" containerName="payload-processing"
Sep 11 18:38:44.451333 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:44.451310    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-791f07"
Sep 11 18:38:44.453886 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:44.453856    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fz6cx\""
Sep 11 18:38:44.456002 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:44.455970    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7842-791f07"]
Sep 11 18:38:44.483492 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:44.483453    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2jdh\" (UniqueName: \"kubernetes.io/projected/8d85c486-e9f0-43c8-b12a-529f00aabe51-kube-api-access-r2jdh\") pod \"test-curl-7842-791f07\" (UID: \"8d85c486-e9f0-43c8-b12a-529f00aabe51\") " pod="opendatahub/test-curl-7842-791f07"
Sep 11 18:38:44.584406 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:44.584366    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2jdh\" (UniqueName: \"kubernetes.io/projected/8d85c486-e9f0-43c8-b12a-529f00aabe51-kube-api-access-r2jdh\") pod \"test-curl-7842-791f07\" (UID: \"8d85c486-e9f0-43c8-b12a-529f00aabe51\") " pod="opendatahub/test-curl-7842-791f07"
Sep 11 18:38:44.593510 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:44.593476    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2jdh\" (UniqueName: \"kubernetes.io/projected/8d85c486-e9f0-43c8-b12a-529f00aabe51-kube-api-access-r2jdh\") pod \"test-curl-7842-791f07\" (UID: \"8d85c486-e9f0-43c8-b12a-529f00aabe51\") " pod="opendatahub/test-curl-7842-791f07"
Sep 11 18:38:44.761273 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:44.761182    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-791f07"
Sep 11 18:38:45.029479 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:45.029449    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7842-791f07"]
Sep 11 18:38:45.032015 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:38:45.031982    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d85c486_e9f0_43c8_b12a_529f00aabe51.slice/crio-ba6fad5f8f237899b7dc8e44d4fc8c6edb724ed9269e37a0acf00b420e287be7 WatchSource:0}: Error finding container ba6fad5f8f237899b7dc8e44d4fc8c6edb724ed9269e37a0acf00b420e287be7: Status 404 returned error can't find the container with id ba6fad5f8f237899b7dc8e44d4fc8c6edb724ed9269e37a0acf00b420e287be7
Sep 11 18:38:45.911364 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:45.911320    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7842-791f07" event={"ID":"8d85c486-e9f0-43c8-b12a-529f00aabe51","Type":"ContainerStarted","Data":"3a6b25feb81774b5403a78d19f435d0801e0dd6a11c1d4b0c30ca2a215d642f4"}
Sep 11 18:38:45.911364 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:45.911364    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7842-791f07" event={"ID":"8d85c486-e9f0-43c8-b12a-529f00aabe51","Type":"ContainerStarted","Data":"ba6fad5f8f237899b7dc8e44d4fc8c6edb724ed9269e37a0acf00b420e287be7"}
Sep 11 18:38:45.925278 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:45.925215    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7842-791f07" podStartSLOduration=1.925197386 podStartE2EDuration="1.925197386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:38:45.923634507 +0000 UTC m=+1724.524995670" watchObservedRunningTime="2026-09-11 18:38:45.925197386 +0000 UTC m=+1724.526558549"
Sep 11 18:38:46.523019 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:46.522973    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7842-791f07"]
Sep 11 18:38:46.527650 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:46.527620    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7842-791f07"]
Sep 11 18:38:46.644058 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:46.643971    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7842-feec1a"]
Sep 11 18:38:46.644330 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:46.644316    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d85c486-e9f0-43c8-b12a-529f00aabe51" containerName="test-curl-7842-791f07"
Sep 11 18:38:46.644376 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:46.644332    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d85c486-e9f0-43c8-b12a-529f00aabe51" containerName="test-curl-7842-791f07"
Sep 11 18:38:46.644412 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:46.644387    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d85c486-e9f0-43c8-b12a-529f00aabe51" containerName="test-curl-7842-791f07"
Sep 11 18:38:46.647312 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:46.647292    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-feec1a"
Sep 11 18:38:46.654785 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:46.654644    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7842-feec1a"]
Sep 11 18:38:46.703461 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:46.703421    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stnvq\" (UniqueName: \"kubernetes.io/projected/bafa12ad-80b3-419e-8684-0b5280bb3b8f-kube-api-access-stnvq\") pod \"test-curl-7842-feec1a\" (UID: \"bafa12ad-80b3-419e-8684-0b5280bb3b8f\") " pod="opendatahub/test-curl-7842-feec1a"
Sep 11 18:38:46.804670 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:46.804626    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stnvq\" (UniqueName: \"kubernetes.io/projected/bafa12ad-80b3-419e-8684-0b5280bb3b8f-kube-api-access-stnvq\") pod \"test-curl-7842-feec1a\" (UID: \"bafa12ad-80b3-419e-8684-0b5280bb3b8f\") " pod="opendatahub/test-curl-7842-feec1a"
Sep 11 18:38:46.813529 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:46.813487    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stnvq\" (UniqueName: \"kubernetes.io/projected/bafa12ad-80b3-419e-8684-0b5280bb3b8f-kube-api-access-stnvq\") pod \"test-curl-7842-feec1a\" (UID: \"bafa12ad-80b3-419e-8684-0b5280bb3b8f\") " pod="opendatahub/test-curl-7842-feec1a"
Sep 11 18:38:46.916824 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:38:46.916739    2576 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7842-791f07" container="test-curl-7842-791f07"
Sep 11 18:38:46.957811 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:46.957770    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-feec1a"
Sep 11 18:38:47.234126 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:47.234090    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7842-feec1a"]
Sep 11 18:38:47.238141 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:38:47.238105    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbafa12ad_80b3_419e_8684_0b5280bb3b8f.slice/crio-c56801f7fda318e8130071c16aaf8542e6a34748971da4415883e0387bd32f6a WatchSource:0}: Error finding container c56801f7fda318e8130071c16aaf8542e6a34748971da4415883e0387bd32f6a: Status 404 returned error can't find the container with id c56801f7fda318e8130071c16aaf8542e6a34748971da4415883e0387bd32f6a
Sep 11 18:38:47.923153 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:47.923113    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7842-feec1a" event={"ID":"bafa12ad-80b3-419e-8684-0b5280bb3b8f","Type":"ContainerStarted","Data":"466876677e02a0aef59bb948f37c6827e27b8aa8ceb30af7712272be37ebbd1f"}
Sep 11 18:38:47.923640 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:47.923163    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7842-feec1a" event={"ID":"bafa12ad-80b3-419e-8684-0b5280bb3b8f","Type":"ContainerStarted","Data":"c56801f7fda318e8130071c16aaf8542e6a34748971da4415883e0387bd32f6a"}
Sep 11 18:38:47.923640 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:47.923256    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7842-791f07" podUID="8d85c486-e9f0-43c8-b12a-529f00aabe51" containerName="test-curl-7842-791f07" containerID="cri-o://3a6b25feb81774b5403a78d19f435d0801e0dd6a11c1d4b0c30ca2a215d642f4" gracePeriod=2
Sep 11 18:38:47.940250 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:47.940193    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7842-feec1a" podStartSLOduration=1.940165843 podStartE2EDuration="1.940165843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:38:47.938384812 +0000 UTC m=+1726.539745976" watchObservedRunningTime="2026-09-11 18:38:47.940165843 +0000 UTC m=+1726.541527040"
Sep 11 18:38:48.506980 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:48.506940    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7842-feec1a"]
Sep 11 18:38:48.509746 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:48.509716    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7842-feec1a"]
Sep 11 18:38:48.929649 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:38:48.929609    2576 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7842-feec1a" container="test-curl-7842-feec1a"
Sep 11 18:38:49.935628 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:49.935538    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7842-feec1a" podUID="bafa12ad-80b3-419e-8684-0b5280bb3b8f" containerName="test-curl-7842-feec1a" containerID="cri-o://466876677e02a0aef59bb948f37c6827e27b8aa8ceb30af7712272be37ebbd1f" gracePeriod=2
Sep 11 18:38:50.099482 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:50.099454    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-791f07"
Sep 11 18:38:50.236104 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:50.236063    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2jdh\" (UniqueName: \"kubernetes.io/projected/8d85c486-e9f0-43c8-b12a-529f00aabe51-kube-api-access-r2jdh\") pod \"8d85c486-e9f0-43c8-b12a-529f00aabe51\" (UID: \"8d85c486-e9f0-43c8-b12a-529f00aabe51\") "
Sep 11 18:38:50.238198 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:50.238174    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d85c486-e9f0-43c8-b12a-529f00aabe51-kube-api-access-r2jdh" (OuterVolumeSpecName: "kube-api-access-r2jdh") pod "8d85c486-e9f0-43c8-b12a-529f00aabe51" (UID: "8d85c486-e9f0-43c8-b12a-529f00aabe51"). InnerVolumeSpecName "kube-api-access-r2jdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:38:50.336824 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:50.336778    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2jdh\" (UniqueName: \"kubernetes.io/projected/8d85c486-e9f0-43c8-b12a-529f00aabe51-kube-api-access-r2jdh\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:38:50.491028 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:50.490944    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"]
Sep 11 18:38:50.941954 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:50.941925    2576 generic.go:356] "Generic (PLEG): container finished" podID="8d85c486-e9f0-43c8-b12a-529f00aabe51" containerID="3a6b25feb81774b5403a78d19f435d0801e0dd6a11c1d4b0c30ca2a215d642f4" exitCode=137
Sep 11 18:38:50.942338 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:50.941970    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-791f07"
Sep 11 18:38:50.942338 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:50.942030    2576 scope.go:117] "RemoveContainer" containerID="3a6b25feb81774b5403a78d19f435d0801e0dd6a11c1d4b0c30ca2a215d642f4"
Sep 11 18:38:50.953825 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:50.953794    2576 scope.go:117] "RemoveContainer" containerID="3a6b25feb81774b5403a78d19f435d0801e0dd6a11c1d4b0c30ca2a215d642f4"
Sep 11 18:38:50.954180 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:38:50.954158    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a6b25feb81774b5403a78d19f435d0801e0dd6a11c1d4b0c30ca2a215d642f4\": container with ID starting with 3a6b25feb81774b5403a78d19f435d0801e0dd6a11c1d4b0c30ca2a215d642f4 not found: ID does not exist" containerID="3a6b25feb81774b5403a78d19f435d0801e0dd6a11c1d4b0c30ca2a215d642f4"
Sep 11 18:38:50.954249 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:50.954189    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6b25feb81774b5403a78d19f435d0801e0dd6a11c1d4b0c30ca2a215d642f4"} err="failed to get container status \"3a6b25feb81774b5403a78d19f435d0801e0dd6a11c1d4b0c30ca2a215d642f4\": rpc error: code = NotFound desc = could not find container \"3a6b25feb81774b5403a78d19f435d0801e0dd6a11c1d4b0c30ca2a215d642f4\": container with ID starting with 3a6b25feb81774b5403a78d19f435d0801e0dd6a11c1d4b0c30ca2a215d642f4 not found: ID does not exist"
Sep 11 18:38:50.955365 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:38:50.955344    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7842-791f07\" is forbidden: User \"system:node:ip-10-0-134-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-111.ec2.internal' and this object" podUID="8d85c486-e9f0-43c8-b12a-529f00aabe51" pod="opendatahub/test-curl-7842-791f07"
Sep 11 18:38:51.908739 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:38:51.908699    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7842-791f07\" is forbidden: User \"system:node:ip-10-0-134-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-111.ec2.internal' and this object" podUID="8d85c486-e9f0-43c8-b12a-529f00aabe51" pod="opendatahub/test-curl-7842-791f07"
Sep 11 18:38:51.910464 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:51.910440    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d85c486-e9f0-43c8-b12a-529f00aabe51" path="/var/lib/kubelet/pods/8d85c486-e9f0-43c8-b12a-529f00aabe51/volumes"
Sep 11 18:38:52.111361 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:52.111327    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-feec1a"
Sep 11 18:38:52.252868 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:52.252828    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stnvq\" (UniqueName: \"kubernetes.io/projected/bafa12ad-80b3-419e-8684-0b5280bb3b8f-kube-api-access-stnvq\") pod \"bafa12ad-80b3-419e-8684-0b5280bb3b8f\" (UID: \"bafa12ad-80b3-419e-8684-0b5280bb3b8f\") "
Sep 11 18:38:52.254928 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:52.254901    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bafa12ad-80b3-419e-8684-0b5280bb3b8f-kube-api-access-stnvq" (OuterVolumeSpecName: "kube-api-access-stnvq") pod "bafa12ad-80b3-419e-8684-0b5280bb3b8f" (UID: "bafa12ad-80b3-419e-8684-0b5280bb3b8f"). InnerVolumeSpecName "kube-api-access-stnvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:38:52.353570 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:52.353527    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stnvq\" (UniqueName: \"kubernetes.io/projected/bafa12ad-80b3-419e-8684-0b5280bb3b8f-kube-api-access-stnvq\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:38:52.954563 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:52.954525    2576 generic.go:356] "Generic (PLEG): container finished" podID="bafa12ad-80b3-419e-8684-0b5280bb3b8f" containerID="466876677e02a0aef59bb948f37c6827e27b8aa8ceb30af7712272be37ebbd1f" exitCode=137
Sep 11 18:38:52.954563 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:52.954567    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7842-feec1a"
Sep 11 18:38:52.954843 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:52.954640    2576 scope.go:117] "RemoveContainer" containerID="466876677e02a0aef59bb948f37c6827e27b8aa8ceb30af7712272be37ebbd1f"
Sep 11 18:38:52.963924 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:52.963903    2576 scope.go:117] "RemoveContainer" containerID="466876677e02a0aef59bb948f37c6827e27b8aa8ceb30af7712272be37ebbd1f"
Sep 11 18:38:52.964272 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:38:52.964253    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"466876677e02a0aef59bb948f37c6827e27b8aa8ceb30af7712272be37ebbd1f\": container with ID starting with 466876677e02a0aef59bb948f37c6827e27b8aa8ceb30af7712272be37ebbd1f not found: ID does not exist" containerID="466876677e02a0aef59bb948f37c6827e27b8aa8ceb30af7712272be37ebbd1f"
Sep 11 18:38:52.964342 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:52.964282    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"466876677e02a0aef59bb948f37c6827e27b8aa8ceb30af7712272be37ebbd1f"} err="failed to get container status \"466876677e02a0aef59bb948f37c6827e27b8aa8ceb30af7712272be37ebbd1f\": rpc error: code = NotFound desc = could not find container \"466876677e02a0aef59bb948f37c6827e27b8aa8ceb30af7712272be37ebbd1f\": container with ID starting with 466876677e02a0aef59bb948f37c6827e27b8aa8ceb30af7712272be37ebbd1f not found: ID does not exist"
Sep 11 18:38:52.964657 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:38:52.964637    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7842-feec1a\" is forbidden: User \"system:node:ip-10-0-134-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-111.ec2.internal' and this object" podUID="bafa12ad-80b3-419e-8684-0b5280bb3b8f" pod="opendatahub/test-curl-7842-feec1a"
Sep 11 18:38:53.915288 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:53.912699    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bafa12ad-80b3-419e-8684-0b5280bb3b8f" path="/var/lib/kubelet/pods/bafa12ad-80b3-419e-8684-0b5280bb3b8f/volumes"
Sep 11 18:38:55.009085 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:38:55.009047    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:39:02.856221 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:02.856175    2576 scope.go:117] "RemoveContainer" containerID="d6bcc2b80de445fc0941750ec9c7d4f09f102d4aa4d1ef6432b2c7ac4fa606f5"
Sep 11 18:39:05.509927 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:05.509868    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg" podUID="728286c6-8fa6-44f6-9489-f7df2cd59b0b" containerName="main" containerID="cri-o://16d9947e76be6eaee471c38c370c9266e0507f5f5009687db227b1a54a42f946" gracePeriod=45
Sep 11 18:39:05.749271 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:05.749247    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:39:05.878766 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:05.878679    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/728286c6-8fa6-44f6-9489-f7df2cd59b0b-model-cache\") pod \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\" (UID: \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\") "
Sep 11 18:39:05.878766 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:05.878727    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/728286c6-8fa6-44f6-9489-f7df2cd59b0b-tmp-dir\") pod \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\" (UID: \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\") "
Sep 11 18:39:05.878766 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:05.878747    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9p7r\" (UniqueName: \"kubernetes.io/projected/728286c6-8fa6-44f6-9489-f7df2cd59b0b-kube-api-access-w9p7r\") pod \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\" (UID: \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\") "
Sep 11 18:39:05.879045 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:05.878772    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/728286c6-8fa6-44f6-9489-f7df2cd59b0b-home\") pod \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\" (UID: \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\") "
Sep 11 18:39:05.879045 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:05.878899    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/728286c6-8fa6-44f6-9489-f7df2cd59b0b-tls-certs\") pod \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\" (UID: \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\") "
Sep 11 18:39:05.879045 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:05.878940    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/728286c6-8fa6-44f6-9489-f7df2cd59b0b-dshm\") pod \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\" (UID: \"728286c6-8fa6-44f6-9489-f7df2cd59b0b\") "
Sep 11 18:39:05.879045 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:05.879029    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728286c6-8fa6-44f6-9489-f7df2cd59b0b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "728286c6-8fa6-44f6-9489-f7df2cd59b0b" (UID: "728286c6-8fa6-44f6-9489-f7df2cd59b0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:39:05.879045 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:05.879014    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728286c6-8fa6-44f6-9489-f7df2cd59b0b-model-cache" (OuterVolumeSpecName: "model-cache") pod "728286c6-8fa6-44f6-9489-f7df2cd59b0b" (UID: "728286c6-8fa6-44f6-9489-f7df2cd59b0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:39:05.879286 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:05.879052    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728286c6-8fa6-44f6-9489-f7df2cd59b0b-home" (OuterVolumeSpecName: "home") pod "728286c6-8fa6-44f6-9489-f7df2cd59b0b" (UID: "728286c6-8fa6-44f6-9489-f7df2cd59b0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:39:05.879286 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:05.879193    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/728286c6-8fa6-44f6-9489-f7df2cd59b0b-model-cache\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:39:05.879286 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:05.879210    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/728286c6-8fa6-44f6-9489-f7df2cd59b0b-tmp-dir\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:39:05.879286 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:05.879224    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/728286c6-8fa6-44f6-9489-f7df2cd59b0b-home\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:39:05.880932 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:05.880907    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728286c6-8fa6-44f6-9489-f7df2cd59b0b-dshm" (OuterVolumeSpecName: "dshm") pod "728286c6-8fa6-44f6-9489-f7df2cd59b0b" (UID: "728286c6-8fa6-44f6-9489-f7df2cd59b0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:39:05.881031 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:05.880946    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/728286c6-8fa6-44f6-9489-f7df2cd59b0b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "728286c6-8fa6-44f6-9489-f7df2cd59b0b" (UID: "728286c6-8fa6-44f6-9489-f7df2cd59b0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:39:05.881031 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:05.881004    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/728286c6-8fa6-44f6-9489-f7df2cd59b0b-kube-api-access-w9p7r" (OuterVolumeSpecName: "kube-api-access-w9p7r") pod "728286c6-8fa6-44f6-9489-f7df2cd59b0b" (UID: "728286c6-8fa6-44f6-9489-f7df2cd59b0b"). InnerVolumeSpecName "kube-api-access-w9p7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:39:05.980480 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:05.980448    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/728286c6-8fa6-44f6-9489-f7df2cd59b0b-tls-certs\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:39:05.980480 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:05.980477    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/728286c6-8fa6-44f6-9489-f7df2cd59b0b-dshm\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:39:05.980480 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:05.980487    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9p7r\" (UniqueName: \"kubernetes.io/projected/728286c6-8fa6-44f6-9489-f7df2cd59b0b-kube-api-access-w9p7r\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:39:06.023696 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:06.023664    2576 generic.go:356] "Generic (PLEG): container finished" podID="728286c6-8fa6-44f6-9489-f7df2cd59b0b" containerID="16d9947e76be6eaee471c38c370c9266e0507f5f5009687db227b1a54a42f946" exitCode=0
Sep 11 18:39:06.023874 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:06.023719    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"
Sep 11 18:39:06.023874 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:06.023745    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg" event={"ID":"728286c6-8fa6-44f6-9489-f7df2cd59b0b","Type":"ContainerDied","Data":"16d9947e76be6eaee471c38c370c9266e0507f5f5009687db227b1a54a42f946"}
Sep 11 18:39:06.023874 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:06.023796    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg" event={"ID":"728286c6-8fa6-44f6-9489-f7df2cd59b0b","Type":"ContainerDied","Data":"455708a67defdb176b0211012b340c8f81be495406839de35ecbeba16c3b7199"}
Sep 11 18:39:06.023874 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:06.023819    2576 scope.go:117] "RemoveContainer" containerID="16d9947e76be6eaee471c38c370c9266e0507f5f5009687db227b1a54a42f946"
Sep 11 18:39:06.032442 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:06.032422    2576 scope.go:117] "RemoveContainer" containerID="16d9947e76be6eaee471c38c370c9266e0507f5f5009687db227b1a54a42f946"
Sep 11 18:39:06.032724 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:39:06.032705    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d9947e76be6eaee471c38c370c9266e0507f5f5009687db227b1a54a42f946\": container with ID starting with 16d9947e76be6eaee471c38c370c9266e0507f5f5009687db227b1a54a42f946 not found: ID does not exist" containerID="16d9947e76be6eaee471c38c370c9266e0507f5f5009687db227b1a54a42f946"
Sep 11 18:39:06.032781 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:06.032732    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d9947e76be6eaee471c38c370c9266e0507f5f5009687db227b1a54a42f946"} err="failed to get container status \"16d9947e76be6eaee471c38c370c9266e0507f5f5009687db227b1a54a42f946\": rpc error: code = NotFound desc = could not find container \"16d9947e76be6eaee471c38c370c9266e0507f5f5009687db227b1a54a42f946\": container with ID starting with 16d9947e76be6eaee471c38c370c9266e0507f5f5009687db227b1a54a42f946 not found: ID does not exist"
Sep 11 18:39:06.039520 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:06.039495    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"]
Sep 11 18:39:06.043513 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:06.043489    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf9898/sub-test-model-bf9898-kserve-79fb67b5b7-7z5jg"]
Sep 11 18:39:07.910818 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:07.910785    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="728286c6-8fa6-44f6-9489-f7df2cd59b0b" path="/var/lib/kubelet/pods/728286c6-8fa6-44f6-9489-f7df2cd59b0b/volumes"
Sep 11 18:39:25.081011 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.080978    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"]
Sep 11 18:39:25.081486 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.081315    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="728286c6-8fa6-44f6-9489-f7df2cd59b0b" containerName="main"
Sep 11 18:39:25.081486 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.081325    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="728286c6-8fa6-44f6-9489-f7df2cd59b0b" containerName="main"
Sep 11 18:39:25.081486 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.081334    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bafa12ad-80b3-419e-8684-0b5280bb3b8f" containerName="test-curl-7842-feec1a"
Sep 11 18:39:25.081486 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.081340    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bafa12ad-80b3-419e-8684-0b5280bb3b8f" containerName="test-curl-7842-feec1a"
Sep 11 18:39:25.081486 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.081396    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="728286c6-8fa6-44f6-9489-f7df2cd59b0b" containerName="main"
Sep 11 18:39:25.081486 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.081403    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bafa12ad-80b3-419e-8684-0b5280bb3b8f" containerName="test-curl-7842-feec1a"
Sep 11 18:39:25.085972 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.085950    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:39:25.088377 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.088349    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bf9898\"/\"kube-root-ca.crt\""
Sep 11 18:39:25.089166 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.089125    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bf9898\"/\"rate-test-model-bf9898-kserve-self-signed-certs\""
Sep 11 18:39:25.089166 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.089148    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bf9898\"/\"openshift-service-ca.crt\""
Sep 11 18:39:25.089166 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.089152    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bf9898\"/\"default-dockercfg-b87pn\""
Sep 11 18:39:25.095304 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.095272    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"]
Sep 11 18:39:25.131957 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.131921    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-model-cache\") pod \"rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd\" (UID: \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:39:25.131957 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.131966    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-dshm\") pod \"rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd\" (UID: \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:39:25.132173 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.131996    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-tmp-dir\") pod \"rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd\" (UID: \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:39:25.132173 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.132058    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-tls-certs\") pod \"rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd\" (UID: \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:39:25.132173 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.132079    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q848d\" (UniqueName: \"kubernetes.io/projected/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-kube-api-access-q848d\") pod \"rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd\" (UID: \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:39:25.132173 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.132112    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-home\") pod \"rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd\" (UID: \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:39:25.233133 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.233095    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-dshm\") pod \"rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd\" (UID: \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:39:25.233133 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.233137    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-tmp-dir\") pod \"rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd\" (UID: \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:39:25.233391 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.233166    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-tls-certs\") pod \"rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd\" (UID: \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:39:25.233391 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.233186    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q848d\" (UniqueName: \"kubernetes.io/projected/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-kube-api-access-q848d\") pod \"rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd\" (UID: \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:39:25.233391 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.233227    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-home\") pod \"rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd\" (UID: \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:39:25.233391 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.233260    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-model-cache\") pod \"rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd\" (UID: \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:39:25.233694 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.233673    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-model-cache\") pod \"rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd\" (UID: \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:39:25.233804 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.233677    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-tmp-dir\") pod \"rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd\" (UID: \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:39:25.233804 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.233751    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-home\") pod \"rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd\" (UID: \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:39:25.235646 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.235617    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-dshm\") pod \"rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd\" (UID: \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:39:25.235753 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.235738    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-tls-certs\") pod \"rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd\" (UID: \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:39:25.241732 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.241702    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q848d\" (UniqueName: \"kubernetes.io/projected/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-kube-api-access-q848d\") pod \"rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd\" (UID: \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\") " pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:39:25.396430 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.396320    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:39:25.670931 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:25.670766    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"]
Sep 11 18:39:25.673653 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:39:25.673622    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c994ed0_7e79_48ee_b4c2_e0101e12efdc.slice/crio-6d89baacbe4dbf51f49af835b2eddd6477b357c0e5801e5ad6063800b8d6ac22 WatchSource:0}: Error finding container 6d89baacbe4dbf51f49af835b2eddd6477b357c0e5801e5ad6063800b8d6ac22: Status 404 returned error can't find the container with id 6d89baacbe4dbf51f49af835b2eddd6477b357c0e5801e5ad6063800b8d6ac22
Sep 11 18:39:26.124940 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:26.124905    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd" event={"ID":"4c994ed0-7e79-48ee-b4c2-e0101e12efdc","Type":"ContainerStarted","Data":"85d1a1fdf9e279e360676ce1be39051bf9856edb6975e55649efa1378eac0442"}
Sep 11 18:39:26.125327 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:26.124952    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd" event={"ID":"4c994ed0-7e79-48ee-b4c2-e0101e12efdc","Type":"ContainerStarted","Data":"6d89baacbe4dbf51f49af835b2eddd6477b357c0e5801e5ad6063800b8d6ac22"}
Sep 11 18:39:26.143744 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:26.143681    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd" podStartSLOduration=1.143663565 podStartE2EDuration="1.143663565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:39:26.141768965 +0000 UTC m=+1764.743130129" watchObservedRunningTime="2026-09-11 18:39:26.143663565 +0000 UTC m=+1764.745024728"
Sep 11 18:39:28.012232 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:28.012190    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:39:35.397037 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:35.396987    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:39:35.397443 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:35.397059    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:39:35.409742 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:35.409708    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:39:36.185786 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:39:36.185753    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:40:02.811448 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:02.811402    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"]
Sep 11 18:40:02.814994 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:02.814969    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:40:02.818319 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:02.818290    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5faf38\"/\"openshift-service-ca.crt\""
Sep 11 18:40:02.818468 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:02.818318    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5faf38\"/\"kube-root-ca.crt\""
Sep 11 18:40:02.818468 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:02.818362    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5faf38\"/\"default-dockercfg-qwk46\""
Sep 11 18:40:02.818468 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:02.818296    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5faf38\"/\"rate-test-model-5faf38-kserve-self-signed-certs\""
Sep 11 18:40:02.824079 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:02.824056    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"]
Sep 11 18:40:02.964820 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:02.964786    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecfd60e8-3547-4669-8da0-26085894bce6-tmp-dir\") pod \"rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz\" (UID: \"ecfd60e8-3547-4669-8da0-26085894bce6\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:40:02.964994 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:02.964828    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecfd60e8-3547-4669-8da0-26085894bce6-home\") pod \"rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz\" (UID: \"ecfd60e8-3547-4669-8da0-26085894bce6\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:40:02.964994 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:02.964850    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecfd60e8-3547-4669-8da0-26085894bce6-dshm\") pod \"rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz\" (UID: \"ecfd60e8-3547-4669-8da0-26085894bce6\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:40:02.964994 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:02.964927    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecfd60e8-3547-4669-8da0-26085894bce6-model-cache\") pod \"rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz\" (UID: \"ecfd60e8-3547-4669-8da0-26085894bce6\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:40:02.964994 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:02.964978    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecfd60e8-3547-4669-8da0-26085894bce6-tls-certs\") pod \"rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz\" (UID: \"ecfd60e8-3547-4669-8da0-26085894bce6\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:40:02.965133 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:02.964998    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csc8g\" (UniqueName: \"kubernetes.io/projected/ecfd60e8-3547-4669-8da0-26085894bce6-kube-api-access-csc8g\") pod \"rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz\" (UID: \"ecfd60e8-3547-4669-8da0-26085894bce6\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:40:03.065818 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:03.065726    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecfd60e8-3547-4669-8da0-26085894bce6-tmp-dir\") pod \"rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz\" (UID: \"ecfd60e8-3547-4669-8da0-26085894bce6\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:40:03.065974 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:03.065880    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecfd60e8-3547-4669-8da0-26085894bce6-home\") pod \"rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz\" (UID: \"ecfd60e8-3547-4669-8da0-26085894bce6\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:40:03.065974 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:03.065923    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecfd60e8-3547-4669-8da0-26085894bce6-dshm\") pod \"rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz\" (UID: \"ecfd60e8-3547-4669-8da0-26085894bce6\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:40:03.065974 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:03.065950    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecfd60e8-3547-4669-8da0-26085894bce6-model-cache\") pod \"rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz\" (UID: \"ecfd60e8-3547-4669-8da0-26085894bce6\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:40:03.066141 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:03.066002    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecfd60e8-3547-4669-8da0-26085894bce6-tls-certs\") pod \"rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz\" (UID: \"ecfd60e8-3547-4669-8da0-26085894bce6\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:40:03.066141 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:03.066024    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csc8g\" (UniqueName: \"kubernetes.io/projected/ecfd60e8-3547-4669-8da0-26085894bce6-kube-api-access-csc8g\") pod \"rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz\" (UID: \"ecfd60e8-3547-4669-8da0-26085894bce6\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:40:03.066257 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:03.066172    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecfd60e8-3547-4669-8da0-26085894bce6-tmp-dir\") pod \"rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz\" (UID: \"ecfd60e8-3547-4669-8da0-26085894bce6\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:40:03.066257 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:03.066244    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecfd60e8-3547-4669-8da0-26085894bce6-home\") pod \"rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz\" (UID: \"ecfd60e8-3547-4669-8da0-26085894bce6\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:40:03.066360 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:03.066306    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecfd60e8-3547-4669-8da0-26085894bce6-model-cache\") pod \"rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz\" (UID: \"ecfd60e8-3547-4669-8da0-26085894bce6\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:40:03.068243 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:03.068224    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecfd60e8-3547-4669-8da0-26085894bce6-dshm\") pod \"rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz\" (UID: \"ecfd60e8-3547-4669-8da0-26085894bce6\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:40:03.068496 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:03.068479    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecfd60e8-3547-4669-8da0-26085894bce6-tls-certs\") pod \"rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz\" (UID: \"ecfd60e8-3547-4669-8da0-26085894bce6\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:40:03.075816 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:03.075787    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csc8g\" (UniqueName: \"kubernetes.io/projected/ecfd60e8-3547-4669-8da0-26085894bce6-kube-api-access-csc8g\") pod \"rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz\" (UID: \"ecfd60e8-3547-4669-8da0-26085894bce6\") " pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:40:03.125816 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:03.125780    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:40:03.402507 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:03.402479    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"]
Sep 11 18:40:03.404909 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:40:03.404884    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecfd60e8_3547_4669_8da0_26085894bce6.slice/crio-20a57ad4355092264b11a65f006b72b934fc6e8f65db456d6f092c304556dcd0 WatchSource:0}: Error finding container 20a57ad4355092264b11a65f006b72b934fc6e8f65db456d6f092c304556dcd0: Status 404 returned error can't find the container with id 20a57ad4355092264b11a65f006b72b934fc6e8f65db456d6f092c304556dcd0
Sep 11 18:40:04.311527 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:04.311490    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz" event={"ID":"ecfd60e8-3547-4669-8da0-26085894bce6","Type":"ContainerStarted","Data":"15ff5dead195a94a3aabe0cb2ae0223ecbac6c3a212ee0656823c457a961258b"}
Sep 11 18:40:04.311527 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:04.311534    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz" event={"ID":"ecfd60e8-3547-4669-8da0-26085894bce6","Type":"ContainerStarted","Data":"20a57ad4355092264b11a65f006b72b934fc6e8f65db456d6f092c304556dcd0"}
Sep 11 18:40:04.330257 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:04.330204    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz" podStartSLOduration=2.330185876 podStartE2EDuration="2.330185876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:40:04.328637089 +0000 UTC m=+1802.929998249" watchObservedRunningTime="2026-09-11 18:40:04.330185876 +0000 UTC m=+1802.931547039"
Sep 11 18:40:06.318136 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:06.318094    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:40:13.126094 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:13.126046    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:40:13.126523 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:13.126110    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:40:13.138893 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:13.138860    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:40:13.367913 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:13.367881    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:40:30.222845 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:30.222808    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:40:39.411552 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:39.411509    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:40:49.212887 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:49.212844    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:40:53.490370 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:53.490331    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"]
Sep 11 18:40:54.429893 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:54.429860    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"]
Sep 11 18:40:58.406127 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:40:58.406082    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:41:00.236599 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:00.236553    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl"]
Sep 11 18:41:00.239832 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:00.239816    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl"
Sep 11 18:41:00.242091 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:00.242064    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-765dd1\""
Sep 11 18:41:00.242216 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:00.242064    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-765dd1-dockercfg-66r8f\""
Sep 11 18:41:00.250558 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:00.250533    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl"]
Sep 11 18:41:00.361955 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:00.361919    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c671cca-ca3a-488e-9235-f61447988a96-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl\" (UID: \"6c671cca-ca3a-488e-9235-f61447988a96\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl"
Sep 11 18:41:00.362113 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:00.361960    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzthd\" (UniqueName: \"kubernetes.io/projected/6c671cca-ca3a-488e-9235-f61447988a96-kube-api-access-hzthd\") pod \"payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl\" (UID: \"6c671cca-ca3a-488e-9235-f61447988a96\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl"
Sep 11 18:41:00.463315 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:00.463276    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c671cca-ca3a-488e-9235-f61447988a96-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl\" (UID: \"6c671cca-ca3a-488e-9235-f61447988a96\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl"
Sep 11 18:41:00.463315 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:00.463317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzthd\" (UniqueName: \"kubernetes.io/projected/6c671cca-ca3a-488e-9235-f61447988a96-kube-api-access-hzthd\") pod \"payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl\" (UID: \"6c671cca-ca3a-488e-9235-f61447988a96\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl"
Sep 11 18:41:00.464069 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:00.464043    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c671cca-ca3a-488e-9235-f61447988a96-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl\" (UID: \"6c671cca-ca3a-488e-9235-f61447988a96\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl"
Sep 11 18:41:00.471427 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:00.471404    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzthd\" (UniqueName: \"kubernetes.io/projected/6c671cca-ca3a-488e-9235-f61447988a96-kube-api-access-hzthd\") pod \"payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl\" (UID: \"6c671cca-ca3a-488e-9235-f61447988a96\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl"
Sep 11 18:41:00.551231 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:00.551159    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl"
Sep 11 18:41:00.824615 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:00.824529    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl"]
Sep 11 18:41:00.828533 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:41:00.828503    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c671cca_ca3a_488e_9235_f61447988a96.slice/crio-620a862b83b3c984e702164c3475ffa5c66afe05f087523fa9a9cfdc8398548b WatchSource:0}: Error finding container 620a862b83b3c984e702164c3475ffa5c66afe05f087523fa9a9cfdc8398548b: Status 404 returned error can't find the container with id 620a862b83b3c984e702164c3475ffa5c66afe05f087523fa9a9cfdc8398548b
Sep 11 18:41:01.584683 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:01.584642    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl" event={"ID":"6c671cca-ca3a-488e-9235-f61447988a96","Type":"ContainerStarted","Data":"d7e85dcde4dea752a9f3feed443abb0668a132dd4fb235dcab9bff366c0e8ebd"}
Sep 11 18:41:01.585063 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:01.584720    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl"
Sep 11 18:41:01.585063 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:01.584745    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl" event={"ID":"6c671cca-ca3a-488e-9235-f61447988a96","Type":"ContainerStarted","Data":"620a862b83b3c984e702164c3475ffa5c66afe05f087523fa9a9cfdc8398548b"}
Sep 11 18:41:01.601717 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:01.601669    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl" podStartSLOduration=1.601653644 podStartE2EDuration="1.601653644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:41:01.600626306 +0000 UTC m=+1860.201987470" watchObservedRunningTime="2026-09-11 18:41:01.601653644 +0000 UTC m=+1860.203014838"
Sep 11 18:41:08.203616 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:08.203567    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:41:08.510119 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:08.510034    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd" podUID="4c994ed0-7e79-48ee-b4c2-e0101e12efdc" containerName="main" containerID="cri-o://85d1a1fdf9e279e360676ce1be39051bf9856edb6975e55649efa1378eac0442" gracePeriod=45
Sep 11 18:41:08.760517 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:08.760457    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:41:08.834060 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:08.834023    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-home\") pod \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\" (UID: \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\") "
Sep 11 18:41:08.834223 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:08.834074    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-tmp-dir\") pod \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\" (UID: \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\") "
Sep 11 18:41:08.834223 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:08.834101    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-tls-certs\") pod \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\" (UID: \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\") "
Sep 11 18:41:08.834223 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:08.834152    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q848d\" (UniqueName: \"kubernetes.io/projected/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-kube-api-access-q848d\") pod \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\" (UID: \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\") "
Sep 11 18:41:08.834223 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:08.834204    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-dshm\") pod \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\" (UID: \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\") "
Sep 11 18:41:08.834472 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:08.834229    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-model-cache\") pod \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\" (UID: \"4c994ed0-7e79-48ee-b4c2-e0101e12efdc\") "
Sep 11 18:41:08.834472 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:08.834307    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4c994ed0-7e79-48ee-b4c2-e0101e12efdc" (UID: "4c994ed0-7e79-48ee-b4c2-e0101e12efdc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:08.834472 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:08.834330    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-home" (OuterVolumeSpecName: "home") pod "4c994ed0-7e79-48ee-b4c2-e0101e12efdc" (UID: "4c994ed0-7e79-48ee-b4c2-e0101e12efdc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:08.834472 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:08.834431    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-home\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:41:08.834472 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:08.834444    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-tmp-dir\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:41:08.834720 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:08.834569    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-model-cache" (OuterVolumeSpecName: "model-cache") pod "4c994ed0-7e79-48ee-b4c2-e0101e12efdc" (UID: "4c994ed0-7e79-48ee-b4c2-e0101e12efdc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:08.836345 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:08.836313    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c994ed0-7e79-48ee-b4c2-e0101e12efdc" (UID: "4c994ed0-7e79-48ee-b4c2-e0101e12efdc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:41:08.836345 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:08.836314    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-dshm" (OuterVolumeSpecName: "dshm") pod "4c994ed0-7e79-48ee-b4c2-e0101e12efdc" (UID: "4c994ed0-7e79-48ee-b4c2-e0101e12efdc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:08.836488 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:08.836370    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-kube-api-access-q848d" (OuterVolumeSpecName: "kube-api-access-q848d") pod "4c994ed0-7e79-48ee-b4c2-e0101e12efdc" (UID: "4c994ed0-7e79-48ee-b4c2-e0101e12efdc"). InnerVolumeSpecName "kube-api-access-q848d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:41:08.935263 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:08.935226    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q848d\" (UniqueName: \"kubernetes.io/projected/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-kube-api-access-q848d\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:41:08.935263 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:08.935255    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-dshm\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:41:08.935263 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:08.935264    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-model-cache\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:41:08.935263 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:08.935273    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c994ed0-7e79-48ee-b4c2-e0101e12efdc-tls-certs\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:41:09.448950 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.448908    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz" podUID="ecfd60e8-3547-4669-8da0-26085894bce6" containerName="main" containerID="cri-o://15ff5dead195a94a3aabe0cb2ae0223ecbac6c3a212ee0656823c457a961258b" gracePeriod=45
Sep 11 18:41:09.622623 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.622565    2576 generic.go:356] "Generic (PLEG): container finished" podID="4c994ed0-7e79-48ee-b4c2-e0101e12efdc" containerID="85d1a1fdf9e279e360676ce1be39051bf9856edb6975e55649efa1378eac0442" exitCode=0
Sep 11 18:41:09.622753 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.622631    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"
Sep 11 18:41:09.622753 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.622631    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd" event={"ID":"4c994ed0-7e79-48ee-b4c2-e0101e12efdc","Type":"ContainerDied","Data":"85d1a1fdf9e279e360676ce1be39051bf9856edb6975e55649efa1378eac0442"}
Sep 11 18:41:09.622753 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.622694    2576 scope.go:117] "RemoveContainer" containerID="85d1a1fdf9e279e360676ce1be39051bf9856edb6975e55649efa1378eac0442"
Sep 11 18:41:09.622972 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.622945    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd" event={"ID":"4c994ed0-7e79-48ee-b4c2-e0101e12efdc","Type":"ContainerDied","Data":"6d89baacbe4dbf51f49af835b2eddd6477b357c0e5801e5ad6063800b8d6ac22"}
Sep 11 18:41:09.624976 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.624954    2576 generic.go:356] "Generic (PLEG): container finished" podID="ecfd60e8-3547-4669-8da0-26085894bce6" containerID="15ff5dead195a94a3aabe0cb2ae0223ecbac6c3a212ee0656823c457a961258b" exitCode=0
Sep 11 18:41:09.625083 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.625032    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz" event={"ID":"ecfd60e8-3547-4669-8da0-26085894bce6","Type":"ContainerDied","Data":"15ff5dead195a94a3aabe0cb2ae0223ecbac6c3a212ee0656823c457a961258b"}
Sep 11 18:41:09.633461 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.633443    2576 scope.go:117] "RemoveContainer" containerID="85d1a1fdf9e279e360676ce1be39051bf9856edb6975e55649efa1378eac0442"
Sep 11 18:41:09.633779 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:41:09.633760    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85d1a1fdf9e279e360676ce1be39051bf9856edb6975e55649efa1378eac0442\": container with ID starting with 85d1a1fdf9e279e360676ce1be39051bf9856edb6975e55649efa1378eac0442 not found: ID does not exist" containerID="85d1a1fdf9e279e360676ce1be39051bf9856edb6975e55649efa1378eac0442"
Sep 11 18:41:09.633849 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.633789    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d1a1fdf9e279e360676ce1be39051bf9856edb6975e55649efa1378eac0442"} err="failed to get container status \"85d1a1fdf9e279e360676ce1be39051bf9856edb6975e55649efa1378eac0442\": rpc error: code = NotFound desc = could not find container \"85d1a1fdf9e279e360676ce1be39051bf9856edb6975e55649efa1378eac0442\": container with ID starting with 85d1a1fdf9e279e360676ce1be39051bf9856edb6975e55649efa1378eac0442 not found: ID does not exist"
Sep 11 18:41:09.649021 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.648998    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"]
Sep 11 18:41:09.652964 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.652939    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf9898/rate-test-model-bf9898-kserve-6bf884cd6c-tqjhd"]
Sep 11 18:41:09.697950 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.697923    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:41:09.742324 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.742251    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecfd60e8-3547-4669-8da0-26085894bce6-tmp-dir\") pod \"ecfd60e8-3547-4669-8da0-26085894bce6\" (UID: \"ecfd60e8-3547-4669-8da0-26085894bce6\") "
Sep 11 18:41:09.742324 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.742302    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecfd60e8-3547-4669-8da0-26085894bce6-dshm\") pod \"ecfd60e8-3547-4669-8da0-26085894bce6\" (UID: \"ecfd60e8-3547-4669-8da0-26085894bce6\") "
Sep 11 18:41:09.742493 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.742338    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csc8g\" (UniqueName: \"kubernetes.io/projected/ecfd60e8-3547-4669-8da0-26085894bce6-kube-api-access-csc8g\") pod \"ecfd60e8-3547-4669-8da0-26085894bce6\" (UID: \"ecfd60e8-3547-4669-8da0-26085894bce6\") "
Sep 11 18:41:09.742493 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.742355    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecfd60e8-3547-4669-8da0-26085894bce6-model-cache\") pod \"ecfd60e8-3547-4669-8da0-26085894bce6\" (UID: \"ecfd60e8-3547-4669-8da0-26085894bce6\") "
Sep 11 18:41:09.742493 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.742372    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecfd60e8-3547-4669-8da0-26085894bce6-tls-certs\") pod \"ecfd60e8-3547-4669-8da0-26085894bce6\" (UID: \"ecfd60e8-3547-4669-8da0-26085894bce6\") "
Sep 11 18:41:09.742493 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.742444    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecfd60e8-3547-4669-8da0-26085894bce6-home\") pod \"ecfd60e8-3547-4669-8da0-26085894bce6\" (UID: \"ecfd60e8-3547-4669-8da0-26085894bce6\") "
Sep 11 18:41:09.742724 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.742536    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecfd60e8-3547-4669-8da0-26085894bce6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ecfd60e8-3547-4669-8da0-26085894bce6" (UID: "ecfd60e8-3547-4669-8da0-26085894bce6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:09.742724 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.742651    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecfd60e8-3547-4669-8da0-26085894bce6-model-cache" (OuterVolumeSpecName: "model-cache") pod "ecfd60e8-3547-4669-8da0-26085894bce6" (UID: "ecfd60e8-3547-4669-8da0-26085894bce6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:09.742857 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.742731    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecfd60e8-3547-4669-8da0-26085894bce6-tmp-dir\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:41:09.742857 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.742750    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecfd60e8-3547-4669-8da0-26085894bce6-model-cache\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:41:09.742857 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.742765    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecfd60e8-3547-4669-8da0-26085894bce6-home" (OuterVolumeSpecName: "home") pod "ecfd60e8-3547-4669-8da0-26085894bce6" (UID: "ecfd60e8-3547-4669-8da0-26085894bce6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:09.744476 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.744455    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecfd60e8-3547-4669-8da0-26085894bce6-kube-api-access-csc8g" (OuterVolumeSpecName: "kube-api-access-csc8g") pod "ecfd60e8-3547-4669-8da0-26085894bce6" (UID: "ecfd60e8-3547-4669-8da0-26085894bce6"). InnerVolumeSpecName "kube-api-access-csc8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:41:09.744574 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.744478    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecfd60e8-3547-4669-8da0-26085894bce6-dshm" (OuterVolumeSpecName: "dshm") pod "ecfd60e8-3547-4669-8da0-26085894bce6" (UID: "ecfd60e8-3547-4669-8da0-26085894bce6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:09.744574 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.744560    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecfd60e8-3547-4669-8da0-26085894bce6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ecfd60e8-3547-4669-8da0-26085894bce6" (UID: "ecfd60e8-3547-4669-8da0-26085894bce6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:41:09.844114 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.844081    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csc8g\" (UniqueName: \"kubernetes.io/projected/ecfd60e8-3547-4669-8da0-26085894bce6-kube-api-access-csc8g\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:41:09.844114 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.844107    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecfd60e8-3547-4669-8da0-26085894bce6-tls-certs\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:41:09.844114 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.844116    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecfd60e8-3547-4669-8da0-26085894bce6-home\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:41:09.844114 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.844123    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecfd60e8-3547-4669-8da0-26085894bce6-dshm\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:41:09.910488 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:09.910456    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c994ed0-7e79-48ee-b4c2-e0101e12efdc" path="/var/lib/kubelet/pods/4c994ed0-7e79-48ee-b4c2-e0101e12efdc/volumes"
Sep 11 18:41:10.630975 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:10.630942    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"
Sep 11 18:41:10.630975 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:10.630967    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz" event={"ID":"ecfd60e8-3547-4669-8da0-26085894bce6","Type":"ContainerDied","Data":"20a57ad4355092264b11a65f006b72b934fc6e8f65db456d6f092c304556dcd0"}
Sep 11 18:41:10.631561 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:10.631020    2576 scope.go:117] "RemoveContainer" containerID="15ff5dead195a94a3aabe0cb2ae0223ecbac6c3a212ee0656823c457a961258b"
Sep 11 18:41:10.645431 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:10.645406    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"]
Sep 11 18:41:10.650982 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:10.650958    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5faf38/rate-test-model-5faf38-kserve-7c85b5cbdc-fv6fz"]
Sep 11 18:41:11.910518 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:11.910474    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecfd60e8-3547-4669-8da0-26085894bce6" path="/var/lib/kubelet/pods/ecfd60e8-3547-4669-8da0-26085894bce6/volumes"
Sep 11 18:41:12.592137 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:12.592105    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl"
Sep 11 18:41:29.552164 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.552086    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf"]
Sep 11 18:41:29.552532 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.552500    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c994ed0-7e79-48ee-b4c2-e0101e12efdc" containerName="main"
Sep 11 18:41:29.552532 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.552515    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c994ed0-7e79-48ee-b4c2-e0101e12efdc" containerName="main"
Sep 11 18:41:29.552532 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.552532    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecfd60e8-3547-4669-8da0-26085894bce6" containerName="main"
Sep 11 18:41:29.552733 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.552537    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecfd60e8-3547-4669-8da0-26085894bce6" containerName="main"
Sep 11 18:41:29.552733 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.552624    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c994ed0-7e79-48ee-b4c2-e0101e12efdc" containerName="main"
Sep 11 18:41:29.552733 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.552632    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecfd60e8-3547-4669-8da0-26085894bce6" containerName="main"
Sep 11 18:41:29.557317 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.557295    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf"
Sep 11 18:41:29.559522 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.559499    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:41:29.560370 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.560342    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 18:41:29.560483 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.560369    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bqwkv\""
Sep 11 18:41:29.560483 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.560433    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 18:41:29.560483 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.560372    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-9c1a4f-serving-cert\""
Sep 11 18:41:29.565626 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.565419    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf"]
Sep 11 18:41:29.715391 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.715355    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c15410c-3316-4c0d-aa9e-44f901cbbf28-metrics-certs\") pod \"maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf\" (UID: \"0c15410c-3316-4c0d-aa9e-44f901cbbf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf"
Sep 11 18:41:29.715573 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.715405    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c15410c-3316-4c0d-aa9e-44f901cbbf28-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf\" (UID: \"0c15410c-3316-4c0d-aa9e-44f901cbbf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf"
Sep 11 18:41:29.715573 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.715465    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96qzw\" (UniqueName: \"kubernetes.io/projected/0c15410c-3316-4c0d-aa9e-44f901cbbf28-kube-api-access-96qzw\") pod \"maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf\" (UID: \"0c15410c-3316-4c0d-aa9e-44f901cbbf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf"
Sep 11 18:41:29.715573 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.715507    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c15410c-3316-4c0d-aa9e-44f901cbbf28-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf\" (UID: \"0c15410c-3316-4c0d-aa9e-44f901cbbf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf"
Sep 11 18:41:29.816909 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.816798    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96qzw\" (UniqueName: \"kubernetes.io/projected/0c15410c-3316-4c0d-aa9e-44f901cbbf28-kube-api-access-96qzw\") pod \"maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf\" (UID: \"0c15410c-3316-4c0d-aa9e-44f901cbbf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf"
Sep 11 18:41:29.816909 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.816848    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c15410c-3316-4c0d-aa9e-44f901cbbf28-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf\" (UID: \"0c15410c-3316-4c0d-aa9e-44f901cbbf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf"
Sep 11 18:41:29.817168 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.816929    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c15410c-3316-4c0d-aa9e-44f901cbbf28-metrics-certs\") pod \"maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf\" (UID: \"0c15410c-3316-4c0d-aa9e-44f901cbbf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf"
Sep 11 18:41:29.817168 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.816976    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c15410c-3316-4c0d-aa9e-44f901cbbf28-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf\" (UID: \"0c15410c-3316-4c0d-aa9e-44f901cbbf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf"
Sep 11 18:41:29.817538 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.817514    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c15410c-3316-4c0d-aa9e-44f901cbbf28-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf\" (UID: \"0c15410c-3316-4c0d-aa9e-44f901cbbf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf"
Sep 11 18:41:29.819428 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.819407    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c15410c-3316-4c0d-aa9e-44f901cbbf28-metrics-certs\") pod \"maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf\" (UID: \"0c15410c-3316-4c0d-aa9e-44f901cbbf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf"
Sep 11 18:41:29.819512 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.819449    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c15410c-3316-4c0d-aa9e-44f901cbbf28-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf\" (UID: \"0c15410c-3316-4c0d-aa9e-44f901cbbf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf"
Sep 11 18:41:29.825210 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.825186    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96qzw\" (UniqueName: \"kubernetes.io/projected/0c15410c-3316-4c0d-aa9e-44f901cbbf28-kube-api-access-96qzw\") pod \"maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf\" (UID: \"0c15410c-3316-4c0d-aa9e-44f901cbbf28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf"
Sep 11 18:41:29.869620 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:29.869565    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf"
Sep 11 18:41:30.143316 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.143229    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z"]
Sep 11 18:41:30.146680 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.146659    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z"
Sep 11 18:41:30.149377 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.149339    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9c1a4f-dockercfg-6v5k9\""
Sep 11 18:41:30.149377 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.149354    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9c1a4f\""
Sep 11 18:41:30.150240 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.150219    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf"]
Sep 11 18:41:30.153386 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:41:30.153361    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c15410c_3316_4c0d_aa9e_44f901cbbf28.slice/crio-60d4273be84abd6e0a4c975ace97aff069386b0675a51c29180c0d00f3bd7d6c WatchSource:0}: Error finding container 60d4273be84abd6e0a4c975ace97aff069386b0675a51c29180c0d00f3bd7d6c: Status 404 returned error can't find the container with id 60d4273be84abd6e0a4c975ace97aff069386b0675a51c29180c0d00f3bd7d6c
Sep 11 18:41:30.154868 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.154847    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:41:30.159029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.159006    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z"]
Sep 11 18:41:30.185125 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.185092    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv"]
Sep 11 18:41:30.188657 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.188638    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv"
Sep 11 18:41:30.198123 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.198094    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv"]
Sep 11 18:41:30.219513 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.219464    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhf9f\" (UniqueName: \"kubernetes.io/projected/3780ecf8-8409-41c0-a946-2f6ff82df614-kube-api-access-dhf9f\") pod \"payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z\" (UID: \"3780ecf8-8409-41c0-a946-2f6ff82df614\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z"
Sep 11 18:41:30.219716 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.219647    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3780ecf8-8409-41c0-a946-2f6ff82df614-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z\" (UID: \"3780ecf8-8409-41c0-a946-2f6ff82df614\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z"
Sep 11 18:41:30.320712 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.320634    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5867f0-0e29-4461-b005-e1388ba981d5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv\" (UID: \"ee5867f0-0e29-4461-b005-e1388ba981d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv"
Sep 11 18:41:30.320934 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.320727    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwjk2\" (UniqueName: \"kubernetes.io/projected/ee5867f0-0e29-4461-b005-e1388ba981d5-kube-api-access-bwjk2\") pod \"payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv\" (UID: \"ee5867f0-0e29-4461-b005-e1388ba981d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv"
Sep 11 18:41:30.320934 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.320781    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3780ecf8-8409-41c0-a946-2f6ff82df614-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z\" (UID: \"3780ecf8-8409-41c0-a946-2f6ff82df614\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z"
Sep 11 18:41:30.320934 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.320838    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhf9f\" (UniqueName: \"kubernetes.io/projected/3780ecf8-8409-41c0-a946-2f6ff82df614-kube-api-access-dhf9f\") pod \"payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z\" (UID: \"3780ecf8-8409-41c0-a946-2f6ff82df614\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z"
Sep 11 18:41:30.321466 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.321444    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3780ecf8-8409-41c0-a946-2f6ff82df614-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z\" (UID: \"3780ecf8-8409-41c0-a946-2f6ff82df614\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z"
Sep 11 18:41:30.329295 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.329261    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhf9f\" (UniqueName: \"kubernetes.io/projected/3780ecf8-8409-41c0-a946-2f6ff82df614-kube-api-access-dhf9f\") pod \"payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z\" (UID: \"3780ecf8-8409-41c0-a946-2f6ff82df614\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z"
Sep 11 18:41:30.422407 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.422312    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5867f0-0e29-4461-b005-e1388ba981d5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv\" (UID: \"ee5867f0-0e29-4461-b005-e1388ba981d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv"
Sep 11 18:41:30.422407 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.422358    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwjk2\" (UniqueName: \"kubernetes.io/projected/ee5867f0-0e29-4461-b005-e1388ba981d5-kube-api-access-bwjk2\") pod \"payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv\" (UID: \"ee5867f0-0e29-4461-b005-e1388ba981d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv"
Sep 11 18:41:30.423131 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.423111    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5867f0-0e29-4461-b005-e1388ba981d5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv\" (UID: \"ee5867f0-0e29-4461-b005-e1388ba981d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv"
Sep 11 18:41:30.430109 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.430080    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwjk2\" (UniqueName: \"kubernetes.io/projected/ee5867f0-0e29-4461-b005-e1388ba981d5-kube-api-access-bwjk2\") pod \"payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv\" (UID: \"ee5867f0-0e29-4461-b005-e1388ba981d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv"
Sep 11 18:41:30.470114 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.470069    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z"
Sep 11 18:41:30.499213 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.499179    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv"
Sep 11 18:41:30.732645 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.732610    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf" event={"ID":"0c15410c-3316-4c0d-aa9e-44f901cbbf28","Type":"ContainerStarted","Data":"a82a2691adb4ac0d64538554599708532de5a67cafa7111fc9afdc2f21ab57bc"}
Sep 11 18:41:30.733038 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.732669    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf" event={"ID":"0c15410c-3316-4c0d-aa9e-44f901cbbf28","Type":"ContainerStarted","Data":"60d4273be84abd6e0a4c975ace97aff069386b0675a51c29180c0d00f3bd7d6c"}
Sep 11 18:41:30.733038 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.732804    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf"
Sep 11 18:41:30.744593 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.744498    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z"]
Sep 11 18:41:30.747097 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:41:30.747064    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3780ecf8_8409_41c0_a946_2f6ff82df614.slice/crio-611686960faf59746f3c1939e7943c2c46b10f3943811004667e7f962ec0eabd WatchSource:0}: Error finding container 611686960faf59746f3c1939e7943c2c46b10f3943811004667e7f962ec0eabd: Status 404 returned error can't find the container with id 611686960faf59746f3c1939e7943c2c46b10f3943811004667e7f962ec0eabd
Sep 11 18:41:30.752144 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.752099    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf" podStartSLOduration=1.2599778289999999 podStartE2EDuration="1.752081789s" totalImagesPullingTime="492.10396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:41:30.751194037 +0000 UTC m=+1889.352555201" watchObservedRunningTime="2026-09-11 18:41:30.752081789 +0000 UTC m=+1889.353442952"
Sep 11 18:41:30.780831 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:30.780794    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv"]
Sep 11 18:41:30.784559 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:41:30.784531    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5867f0_0e29_4461_b005_e1388ba981d5.slice/crio-94d7be31a8eac02a7db4847705c885b5f353d09b0e303f55d527d35567f90777 WatchSource:0}: Error finding container 94d7be31a8eac02a7db4847705c885b5f353d09b0e303f55d527d35567f90777: Status 404 returned error can't find the container with id 94d7be31a8eac02a7db4847705c885b5f353d09b0e303f55d527d35567f90777
Sep 11 18:41:31.739249 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:31.739212    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv" event={"ID":"ee5867f0-0e29-4461-b005-e1388ba981d5","Type":"ContainerStarted","Data":"6d220dd3b5bb3d4e13e3f2483a0636209e7c132edf9c96eec3515dda3e455911"}
Sep 11 18:41:31.739741 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:31.739269    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv" event={"ID":"ee5867f0-0e29-4461-b005-e1388ba981d5","Type":"ContainerStarted","Data":"94d7be31a8eac02a7db4847705c885b5f353d09b0e303f55d527d35567f90777"}
Sep 11 18:41:31.739741 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:31.739334    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv"
Sep 11 18:41:31.740999 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:31.740973    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z" event={"ID":"3780ecf8-8409-41c0-a946-2f6ff82df614","Type":"ContainerStarted","Data":"6c05be2357f0be2946d14f7b9a4823c6d4202fd676f8006d64720dcf12c10e53"}
Sep 11 18:41:31.741129 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:31.741022    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z" event={"ID":"3780ecf8-8409-41c0-a946-2f6ff82df614","Type":"ContainerStarted","Data":"611686960faf59746f3c1939e7943c2c46b10f3943811004667e7f962ec0eabd"}
Sep 11 18:41:31.741129 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:31.741124    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z"
Sep 11 18:41:31.758732 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:31.758674    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv" podStartSLOduration=1.758655374 podStartE2EDuration="1.758655374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:41:31.757097074 +0000 UTC m=+1890.358458239" watchObservedRunningTime="2026-09-11 18:41:31.758655374 +0000 UTC m=+1890.360016536"
Sep 11 18:41:31.775040 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:31.774972    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z" podStartSLOduration=1.77495007 podStartE2EDuration="1.77495007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:41:31.773692246 +0000 UTC m=+1890.375053409" watchObservedRunningTime="2026-09-11 18:41:31.77495007 +0000 UTC m=+1890.376311236"
Sep 11 18:41:34.314631 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:34.314576    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:41:36.746880 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:36.746849    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf"
Sep 11 18:41:42.748050 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:42.748009    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv"
Sep 11 18:41:42.748567 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:42.748104    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z"
Sep 11 18:41:57.132108 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:41:57.132079    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv_ee5867f0-0e29-4461-b005-e1388ba981d5/payload-processing/0.log"
Sep 11 18:42:02.900687 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:42:02.900653    2576 scope.go:117] "RemoveContainer" containerID="81a71313a494b18377c083a6e9329ea9b3c2cb9a2da626a53a10cc7bb321651c"
Sep 11 18:42:12.002668 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:42:12.002613    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:42:31.707731 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:42:31.707696    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:42:38.009942 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:42:38.009894    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl_6c671cca-ca3a-488e-9235-f61447988a96/payload-processing/0.log"
Sep 11 18:42:38.245853 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:42:38.245824    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv_ee5867f0-0e29-4461-b005-e1388ba981d5/payload-processing/0.log"
Sep 11 18:42:45.420246 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:42:45.420208    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4"]
Sep 11 18:42:45.423872 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:42:45.423847    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4"
Sep 11 18:42:45.426621 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:42:45.426596    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-cd2146\""
Sep 11 18:42:45.426741 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:42:45.426658    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-cd2146-dockercfg-9x4vb\""
Sep 11 18:42:45.435361 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:42:45.435333    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4"]
Sep 11 18:42:45.568087 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:42:45.568043    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vv66\" (UniqueName: \"kubernetes.io/projected/a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d-kube-api-access-6vv66\") pod \"payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4\" (UID: \"a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4"
Sep 11 18:42:45.568290 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:42:45.568099    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4\" (UID: \"a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4"
Sep 11 18:42:45.669704 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:42:45.669662    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vv66\" (UniqueName: \"kubernetes.io/projected/a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d-kube-api-access-6vv66\") pod \"payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4\" (UID: \"a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4"
Sep 11 18:42:45.669704 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:42:45.669709    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4\" (UID: \"a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4"
Sep 11 18:42:45.670512 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:42:45.670458    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4\" (UID: \"a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4"
Sep 11 18:42:45.678257 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:42:45.678236    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vv66\" (UniqueName: \"kubernetes.io/projected/a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d-kube-api-access-6vv66\") pod \"payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4\" (UID: \"a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4"
Sep 11 18:42:45.734238 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:42:45.734188    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4"
Sep 11 18:42:46.005908 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:42:46.005880    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4"]
Sep 11 18:42:46.008339 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:42:46.008311    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a6b1c4_f6ed_49b9_8001_cde2b7d6d68d.slice/crio-7f2a9061cf09fa19ddc87ad929bdd9ab6367cdbcefc61bb20caeb262e53fffc4 WatchSource:0}: Error finding container 7f2a9061cf09fa19ddc87ad929bdd9ab6367cdbcefc61bb20caeb262e53fffc4: Status 404 returned error can't find the container with id 7f2a9061cf09fa19ddc87ad929bdd9ab6367cdbcefc61bb20caeb262e53fffc4
Sep 11 18:42:46.117212 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:42:46.117171    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4" event={"ID":"a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d","Type":"ContainerStarted","Data":"d82d908c192860a850662d3016a4c9f8d6c64f59f8286ea94fa09bb1db9749b5"}
Sep 11 18:42:46.117409 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:42:46.117243    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4"
Sep 11 18:42:46.117409 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:42:46.117270    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4" event={"ID":"a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d","Type":"ContainerStarted","Data":"7f2a9061cf09fa19ddc87ad929bdd9ab6367cdbcefc61bb20caeb262e53fffc4"}
Sep 11 18:42:46.134848 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:42:46.134791    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4" podStartSLOduration=1.134773944 podStartE2EDuration="1.134773944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:42:46.132566244 +0000 UTC m=+1964.733927408" watchObservedRunningTime="2026-09-11 18:42:46.134773944 +0000 UTC m=+1964.736135106"
Sep 11 18:42:50.409267 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:42:50.409229    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:42:57.124235 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:42:57.124143    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4"
Sep 11 18:43:02.927781 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:02.927749    2576 scope.go:117] "RemoveContainer" containerID="2aefcba4ca52ec1f8536ca564405e69151f81f29c819ac5de34719cd520e3a28"
Sep 11 18:43:02.936354 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:02.936330    2576 scope.go:117] "RemoveContainer" containerID="ed7cba147b3e89af4008b18d1ad3c6297144a18d37f87b364a535409d6dd684a"
Sep 11 18:43:02.944317 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:02.944295    2576 scope.go:117] "RemoveContainer" containerID="ba1e71cc9a01664c90b765066e2333dfd9612c1b0101d77aefc59177ff9549a3"
Sep 11 18:43:02.951808 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:02.951787    2576 scope.go:117] "RemoveContainer" containerID="f94e02198587ebc65d3ab21e98facc24cedb68718ae6b38b6883a61497a97cb8"
Sep 11 18:43:12.119555 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:12.119519    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2146-4jvdr"]
Sep 11 18:43:12.122930 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:12.122912    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2146-4jvdr"
Sep 11 18:43:12.125073 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:12.125047    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t9kdx\""
Sep 11 18:43:12.130214 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:12.129971    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2146-4jvdr"]
Sep 11 18:43:12.199244 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:12.199205    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8965ee7b-fd4a-4130-92aa-03f9c31166ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-cd2146-4jvdr\" (UID: \"8965ee7b-fd4a-4130-92aa-03f9c31166ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2146-4jvdr"
Sep 11 18:43:12.299923 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:12.299879    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8965ee7b-fd4a-4130-92aa-03f9c31166ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-cd2146-4jvdr\" (UID: \"8965ee7b-fd4a-4130-92aa-03f9c31166ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2146-4jvdr"
Sep 11 18:43:12.300512 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:12.300491    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8965ee7b-fd4a-4130-92aa-03f9c31166ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-cd2146-4jvdr\" (UID: \"8965ee7b-fd4a-4130-92aa-03f9c31166ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2146-4jvdr"
Sep 11 18:43:12.435309 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:12.435196    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2146-4jvdr"
Sep 11 18:43:12.703136 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:12.703111    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2146-4jvdr"]
Sep 11 18:43:12.705179 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:43:12.705151    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8965ee7b_fd4a_4130_92aa_03f9c31166ef.slice/crio-dcf67869c9f97a67ddee85d55033ffeb25e923bb8698d89864ef9c20ab7a5f77 WatchSource:0}: Error finding container dcf67869c9f97a67ddee85d55033ffeb25e923bb8698d89864ef9c20ab7a5f77: Status 404 returned error can't find the container with id dcf67869c9f97a67ddee85d55033ffeb25e923bb8698d89864ef9c20ab7a5f77
Sep 11 18:43:13.249137 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:13.249051    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2146-4jvdr" event={"ID":"8965ee7b-fd4a-4130-92aa-03f9c31166ef","Type":"ContainerStarted","Data":"6e18d9c66dc64cad861e35a5596d15681e8d703996ff23ecb3d93e6695473573"}
Sep 11 18:43:13.249137 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:13.249111    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2146-4jvdr" event={"ID":"8965ee7b-fd4a-4130-92aa-03f9c31166ef","Type":"ContainerStarted","Data":"dcf67869c9f97a67ddee85d55033ffeb25e923bb8698d89864ef9c20ab7a5f77"}
Sep 11 18:43:13.265450 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:13.265398    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2146-4jvdr" podStartSLOduration=1.2653836969999999 podStartE2EDuration="1.265383697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:43:13.263987816 +0000 UTC m=+1991.865348978" watchObservedRunningTime="2026-09-11 18:43:13.265383697 +0000 UTC m=+1991.866744879"
Sep 11 18:43:14.254983 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:14.254949    2576 generic.go:356] "Generic (PLEG): container finished" podID="8965ee7b-fd4a-4130-92aa-03f9c31166ef" containerID="6e18d9c66dc64cad861e35a5596d15681e8d703996ff23ecb3d93e6695473573" exitCode=0
Sep 11 18:43:14.255360 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:14.255032    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2146-4jvdr" event={"ID":"8965ee7b-fd4a-4130-92aa-03f9c31166ef","Type":"ContainerDied","Data":"6e18d9c66dc64cad861e35a5596d15681e8d703996ff23ecb3d93e6695473573"}
Sep 11 18:43:15.390777 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:15.390750    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2146-4jvdr"
Sep 11 18:43:15.529301 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:15.529190    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8965ee7b-fd4a-4130-92aa-03f9c31166ef-maas-api-service-ca\") pod \"8965ee7b-fd4a-4130-92aa-03f9c31166ef\" (UID: \"8965ee7b-fd4a-4130-92aa-03f9c31166ef\") "
Sep 11 18:43:15.529625 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:15.529569    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8965ee7b-fd4a-4130-92aa-03f9c31166ef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8965ee7b-fd4a-4130-92aa-03f9c31166ef" (UID: "8965ee7b-fd4a-4130-92aa-03f9c31166ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:43:15.630148 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:15.630107    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8965ee7b-fd4a-4130-92aa-03f9c31166ef-maas-api-service-ca\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:43:16.264803 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:16.264759    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2146-4jvdr" event={"ID":"8965ee7b-fd4a-4130-92aa-03f9c31166ef","Type":"ContainerDied","Data":"dcf67869c9f97a67ddee85d55033ffeb25e923bb8698d89864ef9c20ab7a5f77"}
Sep 11 18:43:16.264803 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:16.264782    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2146-4jvdr"
Sep 11 18:43:16.265024 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:16.264820    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcf67869c9f97a67ddee85d55033ffeb25e923bb8698d89864ef9c20ab7a5f77"
Sep 11 18:43:23.515846 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:23.515802    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4"]
Sep 11 18:43:23.516364 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:23.516023    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4" podUID="a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d" containerName="payload-pre-processing" containerID="cri-o://d82d908c192860a850662d3016a4c9f8d6c64f59f8286ea94fa09bb1db9749b5" gracePeriod=30
Sep 11 18:43:23.678967 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:23.678935    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2146-4jvdr"]
Sep 11 18:43:23.681714 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:23.681688    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd2146-4jvdr"]
Sep 11 18:43:23.777571 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:23.777504    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4"
Sep 11 18:43:23.902871 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:23.902835    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d-plugins-config-volume\") pod \"a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d\" (UID: \"a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d\") "
Sep 11 18:43:23.903048 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:23.902979    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vv66\" (UniqueName: \"kubernetes.io/projected/a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d-kube-api-access-6vv66\") pod \"a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d\" (UID: \"a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d\") "
Sep 11 18:43:23.903230 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:23.903206    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d" (UID: "a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:43:23.905128 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:23.905103    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d-kube-api-access-6vv66" (OuterVolumeSpecName: "kube-api-access-6vv66") pod "a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d" (UID: "a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d"). InnerVolumeSpecName "kube-api-access-6vv66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:43:23.910960 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:23.910817    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8965ee7b-fd4a-4130-92aa-03f9c31166ef" path="/var/lib/kubelet/pods/8965ee7b-fd4a-4130-92aa-03f9c31166ef/volumes"
Sep 11 18:43:24.003682 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:24.003640    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:43:24.003682 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:24.003672    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vv66\" (UniqueName: \"kubernetes.io/projected/a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d-kube-api-access-6vv66\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:43:24.308380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:24.308339    2576 generic.go:356] "Generic (PLEG): container finished" podID="a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d" containerID="d82d908c192860a850662d3016a4c9f8d6c64f59f8286ea94fa09bb1db9749b5" exitCode=0
Sep 11 18:43:24.308623 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:24.308399    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4"
Sep 11 18:43:24.308623 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:24.308424    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4" event={"ID":"a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d","Type":"ContainerDied","Data":"d82d908c192860a850662d3016a4c9f8d6c64f59f8286ea94fa09bb1db9749b5"}
Sep 11 18:43:24.308623 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:24.308489    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4" event={"ID":"a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d","Type":"ContainerDied","Data":"7f2a9061cf09fa19ddc87ad929bdd9ab6367cdbcefc61bb20caeb262e53fffc4"}
Sep 11 18:43:24.308623 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:24.308519    2576 scope.go:117] "RemoveContainer" containerID="d82d908c192860a850662d3016a4c9f8d6c64f59f8286ea94fa09bb1db9749b5"
Sep 11 18:43:24.317305 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:24.317284    2576 scope.go:117] "RemoveContainer" containerID="d82d908c192860a850662d3016a4c9f8d6c64f59f8286ea94fa09bb1db9749b5"
Sep 11 18:43:24.317596 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:43:24.317567    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d82d908c192860a850662d3016a4c9f8d6c64f59f8286ea94fa09bb1db9749b5\": container with ID starting with d82d908c192860a850662d3016a4c9f8d6c64f59f8286ea94fa09bb1db9749b5 not found: ID does not exist" containerID="d82d908c192860a850662d3016a4c9f8d6c64f59f8286ea94fa09bb1db9749b5"
Sep 11 18:43:24.317684 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:24.317605    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d82d908c192860a850662d3016a4c9f8d6c64f59f8286ea94fa09bb1db9749b5"} err="failed to get container status \"d82d908c192860a850662d3016a4c9f8d6c64f59f8286ea94fa09bb1db9749b5\": rpc error: code = NotFound desc = could not find container \"d82d908c192860a850662d3016a4c9f8d6c64f59f8286ea94fa09bb1db9749b5\": container with ID starting with d82d908c192860a850662d3016a4c9f8d6c64f59f8286ea94fa09bb1db9749b5 not found: ID does not exist"
Sep 11 18:43:24.327215 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:24.327186    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4"]
Sep 11 18:43:24.332245 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:24.332215    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd2146-5cd76bc7db-klgm4"]
Sep 11 18:43:25.910956 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:25.910917    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d" path="/var/lib/kubelet/pods/a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d/volumes"
Sep 11 18:43:28.416255 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:28.416215    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:43:33.842711 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:33.842676    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-765dd1-gsbzj"]
Sep 11 18:43:33.843320 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:33.843005    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8965ee7b-fd4a-4130-92aa-03f9c31166ef" containerName="revoke-keys"
Sep 11 18:43:33.843320 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:33.843016    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8965ee7b-fd4a-4130-92aa-03f9c31166ef" containerName="revoke-keys"
Sep 11 18:43:33.843320 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:33.843028    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d" containerName="payload-pre-processing"
Sep 11 18:43:33.843320 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:33.843033    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d" containerName="payload-pre-processing"
Sep 11 18:43:33.843320 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:33.843101    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8965ee7b-fd4a-4130-92aa-03f9c31166ef" containerName="revoke-keys"
Sep 11 18:43:33.843320 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:33.843109    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1a6b1c4-f6ed-49b9-8001-cde2b7d6d68d" containerName="payload-pre-processing"
Sep 11 18:43:33.847269 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:33.847252    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-765dd1-gsbzj"
Sep 11 18:43:33.849445 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:33.849424    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t9kdx\""
Sep 11 18:43:33.853541 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:33.853518    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-765dd1-gsbzj"]
Sep 11 18:43:33.999251 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:33.999205    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d4d740-8e2b-4981-b4c4-cb69e9b278b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-765dd1-gsbzj\" (UID: \"67d4d740-8e2b-4981-b4c4-cb69e9b278b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-765dd1-gsbzj"
Sep 11 18:43:34.099995 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:34.099908    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d4d740-8e2b-4981-b4c4-cb69e9b278b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-765dd1-gsbzj\" (UID: \"67d4d740-8e2b-4981-b4c4-cb69e9b278b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-765dd1-gsbzj"
Sep 11 18:43:34.100550 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:34.100530    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d4d740-8e2b-4981-b4c4-cb69e9b278b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-765dd1-gsbzj\" (UID: \"67d4d740-8e2b-4981-b4c4-cb69e9b278b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-765dd1-gsbzj"
Sep 11 18:43:34.160015 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:34.159976    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-765dd1-gsbzj"
Sep 11 18:43:34.435866 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:34.435837    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-765dd1-gsbzj"]
Sep 11 18:43:34.437944 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:43:34.437914    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d4d740_8e2b_4981_b4c4_cb69e9b278b9.slice/crio-4a53b7dd2e8702f344d8d40f3476f9394e06ae91e4c51c8ebd494eaadb058500 WatchSource:0}: Error finding container 4a53b7dd2e8702f344d8d40f3476f9394e06ae91e4c51c8ebd494eaadb058500: Status 404 returned error can't find the container with id 4a53b7dd2e8702f344d8d40f3476f9394e06ae91e4c51c8ebd494eaadb058500
Sep 11 18:43:35.361852 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:35.361818    2576 generic.go:356] "Generic (PLEG): container finished" podID="67d4d740-8e2b-4981-b4c4-cb69e9b278b9" containerID="6c97ea4c7294c45a176ef261ddf8b686425aedb17a36a701b077f7d6b48ebbb8" exitCode=0
Sep 11 18:43:35.362228 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:35.361904    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-765dd1-gsbzj" event={"ID":"67d4d740-8e2b-4981-b4c4-cb69e9b278b9","Type":"ContainerDied","Data":"6c97ea4c7294c45a176ef261ddf8b686425aedb17a36a701b077f7d6b48ebbb8"}
Sep 11 18:43:35.362228 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:35.361961    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-765dd1-gsbzj" event={"ID":"67d4d740-8e2b-4981-b4c4-cb69e9b278b9","Type":"ContainerStarted","Data":"4a53b7dd2e8702f344d8d40f3476f9394e06ae91e4c51c8ebd494eaadb058500"}
Sep 11 18:43:36.497315 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:36.497287    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-765dd1-gsbzj"
Sep 11 18:43:36.621175 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:36.621077    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d4d740-8e2b-4981-b4c4-cb69e9b278b9-maas-api-service-ca\") pod \"67d4d740-8e2b-4981-b4c4-cb69e9b278b9\" (UID: \"67d4d740-8e2b-4981-b4c4-cb69e9b278b9\") "
Sep 11 18:43:36.621468 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:36.621443    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d4d740-8e2b-4981-b4c4-cb69e9b278b9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "67d4d740-8e2b-4981-b4c4-cb69e9b278b9" (UID: "67d4d740-8e2b-4981-b4c4-cb69e9b278b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:43:36.722176 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:36.722140    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d4d740-8e2b-4981-b4c4-cb69e9b278b9-maas-api-service-ca\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:43:37.371445 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:37.371410    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-765dd1-gsbzj"
Sep 11 18:43:37.371697 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:37.371405    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-765dd1-gsbzj" event={"ID":"67d4d740-8e2b-4981-b4c4-cb69e9b278b9","Type":"ContainerDied","Data":"4a53b7dd2e8702f344d8d40f3476f9394e06ae91e4c51c8ebd494eaadb058500"}
Sep 11 18:43:37.371697 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:37.371558    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a53b7dd2e8702f344d8d40f3476f9394e06ae91e4c51c8ebd494eaadb058500"
Sep 11 18:43:47.111739 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:47.111702    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:43:49.391078 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:49.391041    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl"]
Sep 11 18:43:49.391540 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:49.391283    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl" podUID="6c671cca-ca3a-488e-9235-f61447988a96" containerName="payload-processing" containerID="cri-o://d7e85dcde4dea752a9f3feed443abb0668a132dd4fb235dcab9bff366c0e8ebd" gracePeriod=30
Sep 11 18:43:49.555698 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:49.555638    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-765dd1-gsbzj"]
Sep 11 18:43:49.557315 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:49.557290    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-765dd1-gsbzj"]
Sep 11 18:43:49.641558 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:49.641498    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl"
Sep 11 18:43:49.718179 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:49.718146    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzthd\" (UniqueName: \"kubernetes.io/projected/6c671cca-ca3a-488e-9235-f61447988a96-kube-api-access-hzthd\") pod \"6c671cca-ca3a-488e-9235-f61447988a96\" (UID: \"6c671cca-ca3a-488e-9235-f61447988a96\") "
Sep 11 18:43:49.718356 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:49.718185    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c671cca-ca3a-488e-9235-f61447988a96-plugins-config-volume\") pod \"6c671cca-ca3a-488e-9235-f61447988a96\" (UID: \"6c671cca-ca3a-488e-9235-f61447988a96\") "
Sep 11 18:43:49.718614 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:49.718574    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c671cca-ca3a-488e-9235-f61447988a96-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6c671cca-ca3a-488e-9235-f61447988a96" (UID: "6c671cca-ca3a-488e-9235-f61447988a96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:43:49.720260 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:49.720232    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c671cca-ca3a-488e-9235-f61447988a96-kube-api-access-hzthd" (OuterVolumeSpecName: "kube-api-access-hzthd") pod "6c671cca-ca3a-488e-9235-f61447988a96" (UID: "6c671cca-ca3a-488e-9235-f61447988a96"). InnerVolumeSpecName "kube-api-access-hzthd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:43:49.819076 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:49.819045    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzthd\" (UniqueName: \"kubernetes.io/projected/6c671cca-ca3a-488e-9235-f61447988a96-kube-api-access-hzthd\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:43:49.819076 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:49.819068    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c671cca-ca3a-488e-9235-f61447988a96-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:43:49.910421 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:49.910328    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67d4d740-8e2b-4981-b4c4-cb69e9b278b9" path="/var/lib/kubelet/pods/67d4d740-8e2b-4981-b4c4-cb69e9b278b9/volumes"
Sep 11 18:43:50.431436 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:50.431399    2576 generic.go:356] "Generic (PLEG): container finished" podID="6c671cca-ca3a-488e-9235-f61447988a96" containerID="d7e85dcde4dea752a9f3feed443abb0668a132dd4fb235dcab9bff366c0e8ebd" exitCode=0
Sep 11 18:43:50.431958 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:50.431451    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl"
Sep 11 18:43:50.431958 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:50.431483    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl" event={"ID":"6c671cca-ca3a-488e-9235-f61447988a96","Type":"ContainerDied","Data":"d7e85dcde4dea752a9f3feed443abb0668a132dd4fb235dcab9bff366c0e8ebd"}
Sep 11 18:43:50.431958 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:50.431536    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl" event={"ID":"6c671cca-ca3a-488e-9235-f61447988a96","Type":"ContainerDied","Data":"620a862b83b3c984e702164c3475ffa5c66afe05f087523fa9a9cfdc8398548b"}
Sep 11 18:43:50.431958 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:50.431559    2576 scope.go:117] "RemoveContainer" containerID="d7e85dcde4dea752a9f3feed443abb0668a132dd4fb235dcab9bff366c0e8ebd"
Sep 11 18:43:50.443184 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:50.443163    2576 scope.go:117] "RemoveContainer" containerID="d7e85dcde4dea752a9f3feed443abb0668a132dd4fb235dcab9bff366c0e8ebd"
Sep 11 18:43:50.443492 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:43:50.443465    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e85dcde4dea752a9f3feed443abb0668a132dd4fb235dcab9bff366c0e8ebd\": container with ID starting with d7e85dcde4dea752a9f3feed443abb0668a132dd4fb235dcab9bff366c0e8ebd not found: ID does not exist" containerID="d7e85dcde4dea752a9f3feed443abb0668a132dd4fb235dcab9bff366c0e8ebd"
Sep 11 18:43:50.443562 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:50.443505    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e85dcde4dea752a9f3feed443abb0668a132dd4fb235dcab9bff366c0e8ebd"} err="failed to get container status \"d7e85dcde4dea752a9f3feed443abb0668a132dd4fb235dcab9bff366c0e8ebd\": rpc error: code = NotFound desc = could not find container \"d7e85dcde4dea752a9f3feed443abb0668a132dd4fb235dcab9bff366c0e8ebd\": container with ID starting with d7e85dcde4dea752a9f3feed443abb0668a132dd4fb235dcab9bff366c0e8ebd not found: ID does not exist"
Sep 11 18:43:50.447214 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:50.447184    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl"]
Sep 11 18:43:50.448882 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:50.448864    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-765dd1-6ddb8fc66c-z4wxl"]
Sep 11 18:43:51.910162 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:43:51.910129    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c671cca-ca3a-488e-9235-f61447988a96" path="/var/lib/kubelet/pods/6c671cca-ca3a-488e-9235-f61447988a96/volumes"
Sep 11 18:44:00.305728 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:00.305691    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:44:02.980015 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:02.979978    2576 scope.go:117] "RemoveContainer" containerID="eca9fdadaab02c4880bfca9478c0ea1dd7ecfcfb5a8d1d81b8ebc1dd159c0bb9"
Sep 11 18:44:50.044141 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.044100    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf"]
Sep 11 18:44:50.044705 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.044368    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf" podUID="0c15410c-3316-4c0d-aa9e-44f901cbbf28" containerName="maas-api" containerID="cri-o://a82a2691adb4ac0d64538554599708532de5a67cafa7111fc9afdc2f21ab57bc" gracePeriod=30
Sep 11 18:44:50.105493 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.105454    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv"]
Sep 11 18:44:50.105730 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.105683    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv" podUID="ee5867f0-0e29-4461-b005-e1388ba981d5" containerName="payload-processing" containerID="cri-o://6d220dd3b5bb3d4e13e3f2483a0636209e7c132edf9c96eec3515dda3e455911" gracePeriod=30
Sep 11 18:44:50.115052 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.115013    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z"]
Sep 11 18:44:50.115276 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.115252    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z" podUID="3780ecf8-8409-41c0-a946-2f6ff82df614" containerName="payload-pre-processing" containerID="cri-o://6c05be2357f0be2946d14f7b9a4823c6d4202fd676f8006d64720dcf12c10e53" gracePeriod=30
Sep 11 18:44:50.384412 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.384382    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf"
Sep 11 18:44:50.396500 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.396409    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv"
Sep 11 18:44:50.409547 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.409520    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z"
Sep 11 18:44:50.426186 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.426153    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c15410c-3316-4c0d-aa9e-44f901cbbf28-openshift-service-ca\") pod \"0c15410c-3316-4c0d-aa9e-44f901cbbf28\" (UID: \"0c15410c-3316-4c0d-aa9e-44f901cbbf28\") "
Sep 11 18:44:50.426355 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.426197    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c15410c-3316-4c0d-aa9e-44f901cbbf28-metrics-certs\") pod \"0c15410c-3316-4c0d-aa9e-44f901cbbf28\" (UID: \"0c15410c-3316-4c0d-aa9e-44f901cbbf28\") "
Sep 11 18:44:50.426355 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.426215    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5867f0-0e29-4461-b005-e1388ba981d5-plugins-config-volume\") pod \"ee5867f0-0e29-4461-b005-e1388ba981d5\" (UID: \"ee5867f0-0e29-4461-b005-e1388ba981d5\") "
Sep 11 18:44:50.426355 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.426248    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c15410c-3316-4c0d-aa9e-44f901cbbf28-maas-api-tls\") pod \"0c15410c-3316-4c0d-aa9e-44f901cbbf28\" (UID: \"0c15410c-3316-4c0d-aa9e-44f901cbbf28\") "
Sep 11 18:44:50.426355 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.426322    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3780ecf8-8409-41c0-a946-2f6ff82df614-plugins-config-volume\") pod \"3780ecf8-8409-41c0-a946-2f6ff82df614\" (UID: \"3780ecf8-8409-41c0-a946-2f6ff82df614\") "
Sep 11 18:44:50.426355 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.426354    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96qzw\" (UniqueName: \"kubernetes.io/projected/0c15410c-3316-4c0d-aa9e-44f901cbbf28-kube-api-access-96qzw\") pod \"0c15410c-3316-4c0d-aa9e-44f901cbbf28\" (UID: \"0c15410c-3316-4c0d-aa9e-44f901cbbf28\") "
Sep 11 18:44:50.426674 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.426380    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhf9f\" (UniqueName: \"kubernetes.io/projected/3780ecf8-8409-41c0-a946-2f6ff82df614-kube-api-access-dhf9f\") pod \"3780ecf8-8409-41c0-a946-2f6ff82df614\" (UID: \"3780ecf8-8409-41c0-a946-2f6ff82df614\") "
Sep 11 18:44:50.426674 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.426412    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwjk2\" (UniqueName: \"kubernetes.io/projected/ee5867f0-0e29-4461-b005-e1388ba981d5-kube-api-access-bwjk2\") pod \"ee5867f0-0e29-4461-b005-e1388ba981d5\" (UID: \"ee5867f0-0e29-4461-b005-e1388ba981d5\") "
Sep 11 18:44:50.426674 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.426647    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c15410c-3316-4c0d-aa9e-44f901cbbf28-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0c15410c-3316-4c0d-aa9e-44f901cbbf28" (UID: "0c15410c-3316-4c0d-aa9e-44f901cbbf28"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:44:50.426829 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.426671    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5867f0-0e29-4461-b005-e1388ba981d5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ee5867f0-0e29-4461-b005-e1388ba981d5" (UID: "ee5867f0-0e29-4461-b005-e1388ba981d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:44:50.426886 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.426838    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3780ecf8-8409-41c0-a946-2f6ff82df614-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3780ecf8-8409-41c0-a946-2f6ff82df614" (UID: "3780ecf8-8409-41c0-a946-2f6ff82df614"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:44:50.428670 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.428626    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c15410c-3316-4c0d-aa9e-44f901cbbf28-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0c15410c-3316-4c0d-aa9e-44f901cbbf28" (UID: "0c15410c-3316-4c0d-aa9e-44f901cbbf28"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:44:50.428825 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.428688    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c15410c-3316-4c0d-aa9e-44f901cbbf28-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0c15410c-3316-4c0d-aa9e-44f901cbbf28" (UID: "0c15410c-3316-4c0d-aa9e-44f901cbbf28"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:44:50.429034 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.429007    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3780ecf8-8409-41c0-a946-2f6ff82df614-kube-api-access-dhf9f" (OuterVolumeSpecName: "kube-api-access-dhf9f") pod "3780ecf8-8409-41c0-a946-2f6ff82df614" (UID: "3780ecf8-8409-41c0-a946-2f6ff82df614"). InnerVolumeSpecName "kube-api-access-dhf9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:44:50.429148 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.429059    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c15410c-3316-4c0d-aa9e-44f901cbbf28-kube-api-access-96qzw" (OuterVolumeSpecName: "kube-api-access-96qzw") pod "0c15410c-3316-4c0d-aa9e-44f901cbbf28" (UID: "0c15410c-3316-4c0d-aa9e-44f901cbbf28"). InnerVolumeSpecName "kube-api-access-96qzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:44:50.429148 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.429078    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5867f0-0e29-4461-b005-e1388ba981d5-kube-api-access-bwjk2" (OuterVolumeSpecName: "kube-api-access-bwjk2") pod "ee5867f0-0e29-4461-b005-e1388ba981d5" (UID: "ee5867f0-0e29-4461-b005-e1388ba981d5"). InnerVolumeSpecName "kube-api-access-bwjk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:44:50.527054 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.527013    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwjk2\" (UniqueName: \"kubernetes.io/projected/ee5867f0-0e29-4461-b005-e1388ba981d5-kube-api-access-bwjk2\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:44:50.527054 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.527048    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c15410c-3316-4c0d-aa9e-44f901cbbf28-openshift-service-ca\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:44:50.527054 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.527058    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c15410c-3316-4c0d-aa9e-44f901cbbf28-metrics-certs\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:44:50.527300 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.527066    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5867f0-0e29-4461-b005-e1388ba981d5-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:44:50.527300 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.527075    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c15410c-3316-4c0d-aa9e-44f901cbbf28-maas-api-tls\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:44:50.527300 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.527084    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3780ecf8-8409-41c0-a946-2f6ff82df614-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:44:50.527300 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.527092    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96qzw\" (UniqueName: \"kubernetes.io/projected/0c15410c-3316-4c0d-aa9e-44f901cbbf28-kube-api-access-96qzw\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:44:50.527300 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.527100    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhf9f\" (UniqueName: \"kubernetes.io/projected/3780ecf8-8409-41c0-a946-2f6ff82df614-kube-api-access-dhf9f\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:44:50.718768 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.718735    2576 generic.go:356] "Generic (PLEG): container finished" podID="0c15410c-3316-4c0d-aa9e-44f901cbbf28" containerID="a82a2691adb4ac0d64538554599708532de5a67cafa7111fc9afdc2f21ab57bc" exitCode=0
Sep 11 18:44:50.718973 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.718785    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf"
Sep 11 18:44:50.718973 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.718831    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf" event={"ID":"0c15410c-3316-4c0d-aa9e-44f901cbbf28","Type":"ContainerDied","Data":"a82a2691adb4ac0d64538554599708532de5a67cafa7111fc9afdc2f21ab57bc"}
Sep 11 18:44:50.718973 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.718885    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf" event={"ID":"0c15410c-3316-4c0d-aa9e-44f901cbbf28","Type":"ContainerDied","Data":"60d4273be84abd6e0a4c975ace97aff069386b0675a51c29180c0d00f3bd7d6c"}
Sep 11 18:44:50.718973 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.718900    2576 scope.go:117] "RemoveContainer" containerID="a82a2691adb4ac0d64538554599708532de5a67cafa7111fc9afdc2f21ab57bc"
Sep 11 18:44:50.720304 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.720283    2576 generic.go:356] "Generic (PLEG): container finished" podID="ee5867f0-0e29-4461-b005-e1388ba981d5" containerID="6d220dd3b5bb3d4e13e3f2483a0636209e7c132edf9c96eec3515dda3e455911" exitCode=0
Sep 11 18:44:50.720400 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.720327    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv"
Sep 11 18:44:50.720400 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.720352    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv" event={"ID":"ee5867f0-0e29-4461-b005-e1388ba981d5","Type":"ContainerDied","Data":"6d220dd3b5bb3d4e13e3f2483a0636209e7c132edf9c96eec3515dda3e455911"}
Sep 11 18:44:50.720400 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.720398    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv" event={"ID":"ee5867f0-0e29-4461-b005-e1388ba981d5","Type":"ContainerDied","Data":"94d7be31a8eac02a7db4847705c885b5f353d09b0e303f55d527d35567f90777"}
Sep 11 18:44:50.722106 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.722081    2576 generic.go:356] "Generic (PLEG): container finished" podID="3780ecf8-8409-41c0-a946-2f6ff82df614" containerID="6c05be2357f0be2946d14f7b9a4823c6d4202fd676f8006d64720dcf12c10e53" exitCode=0
Sep 11 18:44:50.722217 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.722126    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z"
Sep 11 18:44:50.722217 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.722137    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z" event={"ID":"3780ecf8-8409-41c0-a946-2f6ff82df614","Type":"ContainerDied","Data":"6c05be2357f0be2946d14f7b9a4823c6d4202fd676f8006d64720dcf12c10e53"}
Sep 11 18:44:50.722217 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.722177    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z" event={"ID":"3780ecf8-8409-41c0-a946-2f6ff82df614","Type":"ContainerDied","Data":"611686960faf59746f3c1939e7943c2c46b10f3943811004667e7f962ec0eabd"}
Sep 11 18:44:50.728271 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.728250    2576 scope.go:117] "RemoveContainer" containerID="a82a2691adb4ac0d64538554599708532de5a67cafa7111fc9afdc2f21ab57bc"
Sep 11 18:44:50.728646 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:44:50.728615    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a82a2691adb4ac0d64538554599708532de5a67cafa7111fc9afdc2f21ab57bc\": container with ID starting with a82a2691adb4ac0d64538554599708532de5a67cafa7111fc9afdc2f21ab57bc not found: ID does not exist" containerID="a82a2691adb4ac0d64538554599708532de5a67cafa7111fc9afdc2f21ab57bc"
Sep 11 18:44:50.728742 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.728654    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a82a2691adb4ac0d64538554599708532de5a67cafa7111fc9afdc2f21ab57bc"} err="failed to get container status \"a82a2691adb4ac0d64538554599708532de5a67cafa7111fc9afdc2f21ab57bc\": rpc error: code = NotFound desc = could not find container \"a82a2691adb4ac0d64538554599708532de5a67cafa7111fc9afdc2f21ab57bc\": container with ID starting with a82a2691adb4ac0d64538554599708532de5a67cafa7111fc9afdc2f21ab57bc not found: ID does not exist"
Sep 11 18:44:50.728742 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.728671    2576 scope.go:117] "RemoveContainer" containerID="6d220dd3b5bb3d4e13e3f2483a0636209e7c132edf9c96eec3515dda3e455911"
Sep 11 18:44:50.738118 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.738094    2576 scope.go:117] "RemoveContainer" containerID="6d220dd3b5bb3d4e13e3f2483a0636209e7c132edf9c96eec3515dda3e455911"
Sep 11 18:44:50.738499 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:44:50.738475    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d220dd3b5bb3d4e13e3f2483a0636209e7c132edf9c96eec3515dda3e455911\": container with ID starting with 6d220dd3b5bb3d4e13e3f2483a0636209e7c132edf9c96eec3515dda3e455911 not found: ID does not exist" containerID="6d220dd3b5bb3d4e13e3f2483a0636209e7c132edf9c96eec3515dda3e455911"
Sep 11 18:44:50.738619 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.738506    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d220dd3b5bb3d4e13e3f2483a0636209e7c132edf9c96eec3515dda3e455911"} err="failed to get container status \"6d220dd3b5bb3d4e13e3f2483a0636209e7c132edf9c96eec3515dda3e455911\": rpc error: code = NotFound desc = could not find container \"6d220dd3b5bb3d4e13e3f2483a0636209e7c132edf9c96eec3515dda3e455911\": container with ID starting with 6d220dd3b5bb3d4e13e3f2483a0636209e7c132edf9c96eec3515dda3e455911 not found: ID does not exist"
Sep 11 18:44:50.738619 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.738526    2576 scope.go:117] "RemoveContainer" containerID="6c05be2357f0be2946d14f7b9a4823c6d4202fd676f8006d64720dcf12c10e53"
Sep 11 18:44:50.748707 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.748667    2576 scope.go:117] "RemoveContainer" containerID="6c05be2357f0be2946d14f7b9a4823c6d4202fd676f8006d64720dcf12c10e53"
Sep 11 18:44:50.749244 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:44:50.749223    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c05be2357f0be2946d14f7b9a4823c6d4202fd676f8006d64720dcf12c10e53\": container with ID starting with 6c05be2357f0be2946d14f7b9a4823c6d4202fd676f8006d64720dcf12c10e53 not found: ID does not exist" containerID="6c05be2357f0be2946d14f7b9a4823c6d4202fd676f8006d64720dcf12c10e53"
Sep 11 18:44:50.749309 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.749253    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c05be2357f0be2946d14f7b9a4823c6d4202fd676f8006d64720dcf12c10e53"} err="failed to get container status \"6c05be2357f0be2946d14f7b9a4823c6d4202fd676f8006d64720dcf12c10e53\": rpc error: code = NotFound desc = could not find container \"6c05be2357f0be2946d14f7b9a4823c6d4202fd676f8006d64720dcf12c10e53\": container with ID starting with 6c05be2357f0be2946d14f7b9a4823c6d4202fd676f8006d64720dcf12c10e53 not found: ID does not exist"
Sep 11 18:44:50.766212 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.766167    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf"]
Sep 11 18:44:50.767688 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.767666    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9c1a4f-659856f5fb-tsgsf"]
Sep 11 18:44:50.785005 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.784650    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv"]
Sep 11 18:44:50.791744 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.788289    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9c1a4f-6468fbcb47-h4dlv"]
Sep 11 18:44:50.802371 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.802338    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z"]
Sep 11 18:44:50.807356 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:50.807329    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9c1a4f-94fcf4786-d9w8z"]
Sep 11 18:44:51.910903 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:51.910871    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c15410c-3316-4c0d-aa9e-44f901cbbf28" path="/var/lib/kubelet/pods/0c15410c-3316-4c0d-aa9e-44f901cbbf28/volumes"
Sep 11 18:44:51.911294 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:51.911235    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3780ecf8-8409-41c0-a946-2f6ff82df614" path="/var/lib/kubelet/pods/3780ecf8-8409-41c0-a946-2f6ff82df614/volumes"
Sep 11 18:44:51.911688 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:51.911668    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee5867f0-0e29-4461-b005-e1388ba981d5" path="/var/lib/kubelet/pods/ee5867f0-0e29-4461-b005-e1388ba981d5/volumes"
Sep 11 18:44:53.305804 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:53.305766    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:44:57.261549 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.261510    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf9898-fdcgp"]
Sep 11 18:44:57.261960 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.261861    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee5867f0-0e29-4461-b005-e1388ba981d5" containerName="payload-processing"
Sep 11 18:44:57.261960 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.261873    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee5867f0-0e29-4461-b005-e1388ba981d5" containerName="payload-processing"
Sep 11 18:44:57.261960 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.261883    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67d4d740-8e2b-4981-b4c4-cb69e9b278b9" containerName="revoke-keys"
Sep 11 18:44:57.261960 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.261888    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="67d4d740-8e2b-4981-b4c4-cb69e9b278b9" containerName="revoke-keys"
Sep 11 18:44:57.261960 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.261896    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c671cca-ca3a-488e-9235-f61447988a96" containerName="payload-processing"
Sep 11 18:44:57.261960 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.261902    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c671cca-ca3a-488e-9235-f61447988a96" containerName="payload-processing"
Sep 11 18:44:57.261960 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.261908    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3780ecf8-8409-41c0-a946-2f6ff82df614" containerName="payload-pre-processing"
Sep 11 18:44:57.261960 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.261913    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3780ecf8-8409-41c0-a946-2f6ff82df614" containerName="payload-pre-processing"
Sep 11 18:44:57.261960 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.261926    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c15410c-3316-4c0d-aa9e-44f901cbbf28" containerName="maas-api"
Sep 11 18:44:57.261960 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.261931    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c15410c-3316-4c0d-aa9e-44f901cbbf28" containerName="maas-api"
Sep 11 18:44:57.262313 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.261981    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c671cca-ca3a-488e-9235-f61447988a96" containerName="payload-processing"
Sep 11 18:44:57.262313 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.261987    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="67d4d740-8e2b-4981-b4c4-cb69e9b278b9" containerName="revoke-keys"
Sep 11 18:44:57.262313 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.261992    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee5867f0-0e29-4461-b005-e1388ba981d5" containerName="payload-processing"
Sep 11 18:44:57.262313 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.262002    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3780ecf8-8409-41c0-a946-2f6ff82df614" containerName="payload-pre-processing"
Sep 11 18:44:57.262313 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.262007    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c15410c-3316-4c0d-aa9e-44f901cbbf28" containerName="maas-api"
Sep 11 18:44:57.266518 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.266497    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf9898-fdcgp"
Sep 11 18:44:57.268840 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.268814    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:44:57.269005 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.268821    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t9kdx\""
Sep 11 18:44:57.272546 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.272513    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf9898-fdcgp"]
Sep 11 18:44:57.277326 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.277294    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c7ed88c-e706-4062-8cb7-894bf26eee2d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-bf9898-fdcgp\" (UID: \"6c7ed88c-e706-4062-8cb7-894bf26eee2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf9898-fdcgp"
Sep 11 18:44:57.377928 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.377882    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c7ed88c-e706-4062-8cb7-894bf26eee2d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-bf9898-fdcgp\" (UID: \"6c7ed88c-e706-4062-8cb7-894bf26eee2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf9898-fdcgp"
Sep 11 18:44:57.378596 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.378560    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c7ed88c-e706-4062-8cb7-894bf26eee2d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-bf9898-fdcgp\" (UID: \"6c7ed88c-e706-4062-8cb7-894bf26eee2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf9898-fdcgp"
Sep 11 18:44:57.578353 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.578257    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf9898-fdcgp"
Sep 11 18:44:57.850175 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:57.850147    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf9898-fdcgp"]
Sep 11 18:44:57.852797 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:44:57.852767    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c7ed88c_e706_4062_8cb7_894bf26eee2d.slice/crio-8d38d6656a249056de78e3ce48deb3d3d5066c8fb479c590cacbca7cc20a73d6 WatchSource:0}: Error finding container 8d38d6656a249056de78e3ce48deb3d3d5066c8fb479c590cacbca7cc20a73d6: Status 404 returned error can't find the container with id 8d38d6656a249056de78e3ce48deb3d3d5066c8fb479c590cacbca7cc20a73d6
Sep 11 18:44:58.766614 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:58.766569    2576 generic.go:356] "Generic (PLEG): container finished" podID="6c7ed88c-e706-4062-8cb7-894bf26eee2d" containerID="192dab36e06f60498851631746d6cb2588eef903b49c43f7b5afcddb8cb60db0" exitCode=0
Sep 11 18:44:58.767000 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:58.766652    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf9898-fdcgp" event={"ID":"6c7ed88c-e706-4062-8cb7-894bf26eee2d","Type":"ContainerDied","Data":"192dab36e06f60498851631746d6cb2588eef903b49c43f7b5afcddb8cb60db0"}
Sep 11 18:44:58.767000 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:58.766706    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf9898-fdcgp" event={"ID":"6c7ed88c-e706-4062-8cb7-894bf26eee2d","Type":"ContainerStarted","Data":"8d38d6656a249056de78e3ce48deb3d3d5066c8fb479c590cacbca7cc20a73d6"}
Sep 11 18:44:59.903689 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:44:59.903660    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf9898-fdcgp"
Sep 11 18:45:00.000294 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.000258    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c7ed88c-e706-4062-8cb7-894bf26eee2d-maas-api-service-ca\") pod \"6c7ed88c-e706-4062-8cb7-894bf26eee2d\" (UID: \"6c7ed88c-e706-4062-8cb7-894bf26eee2d\") "
Sep 11 18:45:00.000715 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.000690    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c7ed88c-e706-4062-8cb7-894bf26eee2d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6c7ed88c-e706-4062-8cb7-894bf26eee2d" (UID: "6c7ed88c-e706-4062-8cb7-894bf26eee2d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:45:00.101570 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.101478    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c7ed88c-e706-4062-8cb7-894bf26eee2d-maas-api-service-ca\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:45:00.131914 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.131871    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819205-tvc4p"]
Sep 11 18:45:00.132226 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.132212    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c7ed88c-e706-4062-8cb7-894bf26eee2d" containerName="revoke-keys"
Sep 11 18:45:00.132274 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.132227    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c7ed88c-e706-4062-8cb7-894bf26eee2d" containerName="revoke-keys"
Sep 11 18:45:00.132309 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.132290    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c7ed88c-e706-4062-8cb7-894bf26eee2d" containerName="revoke-keys"
Sep 11 18:45:00.135351 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.135323    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5faf38-29819205-xpghn"]
Sep 11 18:45:00.135497 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.135437    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-tvc4p"
Sep 11 18:45:00.138496 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.138470    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5faf38-29819205-xpghn"
Sep 11 18:45:00.144736 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.144671    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819205-tvc4p"]
Sep 11 18:45:00.148228 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.148202    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-tvc4p"
Sep 11 18:45:00.149019 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.148992    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5faf38-29819205-xpghn"]
Sep 11 18:45:00.157476 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.157448    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5faf38-29819205-xpghn"
Sep 11 18:45:00.431201 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.431174    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819205-tvc4p"]
Sep 11 18:45:00.433817 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:45:00.433783    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb13818ad_e848_405b_a8af_be1e9f826728.slice/crio-8d076c37dfdc028b36af4e79592c9004248abbf6637fadeeefd7cc6c29463a20 WatchSource:0}: Error finding container 8d076c37dfdc028b36af4e79592c9004248abbf6637fadeeefd7cc6c29463a20: Status 404 returned error can't find the container with id 8d076c37dfdc028b36af4e79592c9004248abbf6637fadeeefd7cc6c29463a20
Sep 11 18:45:00.438136 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.438075    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5faf38-29819205-xpghn"]
Sep 11 18:45:00.440755 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:45:00.440730    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1d9dff_d29f_483e_a140_44f2692080ce.slice/crio-e55391f18893d83da86a65dfa76126fb3982e334e4c7caf344ce0fd3db7a6f36 WatchSource:0}: Error finding container e55391f18893d83da86a65dfa76126fb3982e334e4c7caf344ce0fd3db7a6f36: Status 404 returned error can't find the container with id e55391f18893d83da86a65dfa76126fb3982e334e4c7caf344ce0fd3db7a6f36
Sep 11 18:45:00.778348 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.778313    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5faf38-29819205-xpghn" event={"ID":"8b1d9dff-d29f-483e-a140-44f2692080ce","Type":"ContainerStarted","Data":"198bfe70c2c92207d9360943d507d96c5f2225d166c2ac8244d11a92ea15e7a5"}
Sep 11 18:45:00.778554 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.778370    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5faf38-29819205-xpghn" event={"ID":"8b1d9dff-d29f-483e-a140-44f2692080ce","Type":"ContainerStarted","Data":"e55391f18893d83da86a65dfa76126fb3982e334e4c7caf344ce0fd3db7a6f36"}
Sep 11 18:45:00.779709 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.779689    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf9898-fdcgp"
Sep 11 18:45:00.779815 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.779706    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf9898-fdcgp" event={"ID":"6c7ed88c-e706-4062-8cb7-894bf26eee2d","Type":"ContainerDied","Data":"8d38d6656a249056de78e3ce48deb3d3d5066c8fb479c590cacbca7cc20a73d6"}
Sep 11 18:45:00.779815 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.779748    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d38d6656a249056de78e3ce48deb3d3d5066c8fb479c590cacbca7cc20a73d6"
Sep 11 18:45:00.781411 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.781385    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-tvc4p" event={"ID":"b13818ad-e848-405b-a8af-be1e9f826728","Type":"ContainerStarted","Data":"d21f0cd8cb5d2388a744c871a4eb674a73d6dfee6d8c1eb66aebf3288c887dd4"}
Sep 11 18:45:00.781525 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.781427    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-tvc4p" event={"ID":"b13818ad-e848-405b-a8af-be1e9f826728","Type":"ContainerStarted","Data":"8d076c37dfdc028b36af4e79592c9004248abbf6637fadeeefd7cc6c29463a20"}
Sep 11 18:45:00.794841 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.794778    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5faf38-29819205-xpghn" podStartSLOduration=0.794759541 podStartE2EDuration="794.759541ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:45:00.79296253 +0000 UTC m=+2099.394323694" watchObservedRunningTime="2026-09-11 18:45:00.794759541 +0000 UTC m=+2099.396120704"
Sep 11 18:45:00.808825 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:00.808768    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-tvc4p" podStartSLOduration=0.808749822 podStartE2EDuration="808.749822ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:45:00.807750777 +0000 UTC m=+2099.409111967" watchObservedRunningTime="2026-09-11 18:45:00.808749822 +0000 UTC m=+2099.410110985"
Sep 11 18:45:01.790320 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:01.790288    2576 generic.go:356] "Generic (PLEG): container finished" podID="b13818ad-e848-405b-a8af-be1e9f826728" containerID="d21f0cd8cb5d2388a744c871a4eb674a73d6dfee6d8c1eb66aebf3288c887dd4" exitCode=0
Sep 11 18:45:01.790763 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:01.790365    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-tvc4p" event={"ID":"b13818ad-e848-405b-a8af-be1e9f826728","Type":"ContainerDied","Data":"d21f0cd8cb5d2388a744c871a4eb674a73d6dfee6d8c1eb66aebf3288c887dd4"}
Sep 11 18:45:01.791872 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:01.791849    2576 generic.go:356] "Generic (PLEG): container finished" podID="8b1d9dff-d29f-483e-a140-44f2692080ce" containerID="198bfe70c2c92207d9360943d507d96c5f2225d166c2ac8244d11a92ea15e7a5" exitCode=0
Sep 11 18:45:01.791966 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:01.791908    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5faf38-29819205-xpghn" event={"ID":"8b1d9dff-d29f-483e-a140-44f2692080ce","Type":"ContainerDied","Data":"198bfe70c2c92207d9360943d507d96c5f2225d166c2ac8244d11a92ea15e7a5"}
Sep 11 18:45:02.930894 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:02.930867    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-tvc4p"
Sep 11 18:45:02.966550 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:02.966522    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5faf38-29819205-xpghn"
Sep 11 18:45:03.802414 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:03.802382    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-tvc4p"
Sep 11 18:45:03.802414 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:03.802390    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-tvc4p" event={"ID":"b13818ad-e848-405b-a8af-be1e9f826728","Type":"ContainerDied","Data":"8d076c37dfdc028b36af4e79592c9004248abbf6637fadeeefd7cc6c29463a20"}
Sep 11 18:45:03.802708 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:03.802442    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d076c37dfdc028b36af4e79592c9004248abbf6637fadeeefd7cc6c29463a20"
Sep 11 18:45:03.803743 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:03.803724    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5faf38-29819205-xpghn"
Sep 11 18:45:03.803863 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:03.803768    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5faf38-29819205-xpghn" event={"ID":"8b1d9dff-d29f-483e-a140-44f2692080ce","Type":"ContainerDied","Data":"e55391f18893d83da86a65dfa76126fb3982e334e4c7caf344ce0fd3db7a6f36"}
Sep 11 18:45:03.803863 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:03.803809    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e55391f18893d83da86a65dfa76126fb3982e334e4c7caf344ce0fd3db7a6f36"
Sep 11 18:45:07.877544 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:07.877500    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx"]
Sep 11 18:45:07.877995 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:07.877753    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx" podUID="1e4fb425-8f69-41f9-8eab-33a44f90a28a" containerName="payload-pre-processing" containerID="cri-o://e12190c1d10ebe3297b9a00f0b56d8b5dc5e41f742445b94a0d8aad26248ba00" gracePeriod=30
Sep 11 18:45:08.035738 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:08.035701    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf9898-fdcgp"]
Sep 11 18:45:08.039412 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:08.039373    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf9898-fdcgp"]
Sep 11 18:45:08.137846 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:08.137777    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx"
Sep 11 18:45:08.175780 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:08.175737    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e4fb425-8f69-41f9-8eab-33a44f90a28a-plugins-config-volume\") pod \"1e4fb425-8f69-41f9-8eab-33a44f90a28a\" (UID: \"1e4fb425-8f69-41f9-8eab-33a44f90a28a\") "
Sep 11 18:45:08.175780 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:08.175781    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52m4c\" (UniqueName: \"kubernetes.io/projected/1e4fb425-8f69-41f9-8eab-33a44f90a28a-kube-api-access-52m4c\") pod \"1e4fb425-8f69-41f9-8eab-33a44f90a28a\" (UID: \"1e4fb425-8f69-41f9-8eab-33a44f90a28a\") "
Sep 11 18:45:08.176134 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:08.176106    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e4fb425-8f69-41f9-8eab-33a44f90a28a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1e4fb425-8f69-41f9-8eab-33a44f90a28a" (UID: "1e4fb425-8f69-41f9-8eab-33a44f90a28a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:45:08.177999 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:08.177970    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e4fb425-8f69-41f9-8eab-33a44f90a28a-kube-api-access-52m4c" (OuterVolumeSpecName: "kube-api-access-52m4c") pod "1e4fb425-8f69-41f9-8eab-33a44f90a28a" (UID: "1e4fb425-8f69-41f9-8eab-33a44f90a28a"). InnerVolumeSpecName "kube-api-access-52m4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:45:08.276448 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:08.276409    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e4fb425-8f69-41f9-8eab-33a44f90a28a-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:45:08.276448 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:08.276441    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52m4c\" (UniqueName: \"kubernetes.io/projected/1e4fb425-8f69-41f9-8eab-33a44f90a28a-kube-api-access-52m4c\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:45:08.829707 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:08.829672    2576 generic.go:356] "Generic (PLEG): container finished" podID="1e4fb425-8f69-41f9-8eab-33a44f90a28a" containerID="e12190c1d10ebe3297b9a00f0b56d8b5dc5e41f742445b94a0d8aad26248ba00" exitCode=0
Sep 11 18:45:08.829875 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:08.829722    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx"
Sep 11 18:45:08.829875 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:08.829758    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx" event={"ID":"1e4fb425-8f69-41f9-8eab-33a44f90a28a","Type":"ContainerDied","Data":"e12190c1d10ebe3297b9a00f0b56d8b5dc5e41f742445b94a0d8aad26248ba00"}
Sep 11 18:45:08.829875 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:08.829811    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx" event={"ID":"1e4fb425-8f69-41f9-8eab-33a44f90a28a","Type":"ContainerDied","Data":"95fc20a36ed8ad8be8f5d34300de66d685bdb41901e18be58ed517d021fdcb92"}
Sep 11 18:45:08.829875 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:08.829835    2576 scope.go:117] "RemoveContainer" containerID="e12190c1d10ebe3297b9a00f0b56d8b5dc5e41f742445b94a0d8aad26248ba00"
Sep 11 18:45:08.839958 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:08.839935    2576 scope.go:117] "RemoveContainer" containerID="e12190c1d10ebe3297b9a00f0b56d8b5dc5e41f742445b94a0d8aad26248ba00"
Sep 11 18:45:08.840262 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:45:08.840244    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e12190c1d10ebe3297b9a00f0b56d8b5dc5e41f742445b94a0d8aad26248ba00\": container with ID starting with e12190c1d10ebe3297b9a00f0b56d8b5dc5e41f742445b94a0d8aad26248ba00 not found: ID does not exist" containerID="e12190c1d10ebe3297b9a00f0b56d8b5dc5e41f742445b94a0d8aad26248ba00"
Sep 11 18:45:08.840324 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:08.840270    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e12190c1d10ebe3297b9a00f0b56d8b5dc5e41f742445b94a0d8aad26248ba00"} err="failed to get container status \"e12190c1d10ebe3297b9a00f0b56d8b5dc5e41f742445b94a0d8aad26248ba00\": rpc error: code = NotFound desc = could not find container \"e12190c1d10ebe3297b9a00f0b56d8b5dc5e41f742445b94a0d8aad26248ba00\": container with ID starting with e12190c1d10ebe3297b9a00f0b56d8b5dc5e41f742445b94a0d8aad26248ba00 not found: ID does not exist"
Sep 11 18:45:08.858399 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:08.858361    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx"]
Sep 11 18:45:08.872309 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:08.872276    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf9898-54ff665d66-ltkvx"]
Sep 11 18:45:09.909816 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:09.909782    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e4fb425-8f69-41f9-8eab-33a44f90a28a" path="/var/lib/kubelet/pods/1e4fb425-8f69-41f9-8eab-33a44f90a28a/volumes"
Sep 11 18:45:09.910182 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:09.910121    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c7ed88c-e706-4062-8cb7-894bf26eee2d" path="/var/lib/kubelet/pods/6c7ed88c-e706-4062-8cb7-894bf26eee2d/volumes"
Sep 11 18:45:10.706477 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:10.706440    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:45:15.102696 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:15.102661    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5faf38-wdc46"]
Sep 11 18:45:15.103174 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:15.102973    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b13818ad-e848-405b-a8af-be1e9f826728" containerName="cleanup"
Sep 11 18:45:15.103174 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:15.102984    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b13818ad-e848-405b-a8af-be1e9f826728" containerName="cleanup"
Sep 11 18:45:15.103174 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:15.102996    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b1d9dff-d29f-483e-a140-44f2692080ce" containerName="cleanup"
Sep 11 18:45:15.103174 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:15.103002    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b1d9dff-d29f-483e-a140-44f2692080ce" containerName="cleanup"
Sep 11 18:45:15.103174 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:15.103010    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e4fb425-8f69-41f9-8eab-33a44f90a28a" containerName="payload-pre-processing"
Sep 11 18:45:15.103174 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:15.103017    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e4fb425-8f69-41f9-8eab-33a44f90a28a" containerName="payload-pre-processing"
Sep 11 18:45:15.103174 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:15.103102    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e4fb425-8f69-41f9-8eab-33a44f90a28a" containerName="payload-pre-processing"
Sep 11 18:45:15.103174 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:15.103113    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b1d9dff-d29f-483e-a140-44f2692080ce" containerName="cleanup"
Sep 11 18:45:15.103174 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:15.103120    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b13818ad-e848-405b-a8af-be1e9f826728" containerName="cleanup"
Sep 11 18:45:15.107242 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:15.107222    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5faf38-wdc46"
Sep 11 18:45:15.109448 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:15.109426    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:45:15.109448 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:15.109433    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t9kdx\""
Sep 11 18:45:15.113185 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:15.113153    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5faf38-wdc46"]
Sep 11 18:45:15.239761 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:15.239725    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f67854-2a93-4428-a8b0-7534285b7b35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-5faf38-wdc46\" (UID: \"84f67854-2a93-4428-a8b0-7534285b7b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5faf38-wdc46"
Sep 11 18:45:15.340412 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:15.340376    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f67854-2a93-4428-a8b0-7534285b7b35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-5faf38-wdc46\" (UID: \"84f67854-2a93-4428-a8b0-7534285b7b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5faf38-wdc46"
Sep 11 18:45:15.341025 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:15.341002    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f67854-2a93-4428-a8b0-7534285b7b35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-5faf38-wdc46\" (UID: \"84f67854-2a93-4428-a8b0-7534285b7b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5faf38-wdc46"
Sep 11 18:45:15.419575 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:15.419473    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5faf38-wdc46"
Sep 11 18:45:15.700743 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:15.700713    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5faf38-wdc46"]
Sep 11 18:45:15.702807 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:45:15.702778    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f67854_2a93_4428_a8b0_7534285b7b35.slice/crio-369b116e94d7b144af7e95379ad95cca89d1dd8c3ee22c5dfbaf1b0a9a9b957f WatchSource:0}: Error finding container 369b116e94d7b144af7e95379ad95cca89d1dd8c3ee22c5dfbaf1b0a9a9b957f: Status 404 returned error can't find the container with id 369b116e94d7b144af7e95379ad95cca89d1dd8c3ee22c5dfbaf1b0a9a9b957f
Sep 11 18:45:15.863779 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:15.863745    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5faf38-wdc46" event={"ID":"84f67854-2a93-4428-a8b0-7534285b7b35","Type":"ContainerStarted","Data":"369b116e94d7b144af7e95379ad95cca89d1dd8c3ee22c5dfbaf1b0a9a9b957f"}
Sep 11 18:45:16.870131 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:16.870095    2576 generic.go:356] "Generic (PLEG): container finished" podID="84f67854-2a93-4428-a8b0-7534285b7b35" containerID="db087feb6260250c021b57e893c12354ddf4c74fee6c40d3783a39c4117a48f7" exitCode=0
Sep 11 18:45:16.870515 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:16.870184    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5faf38-wdc46" event={"ID":"84f67854-2a93-4428-a8b0-7534285b7b35","Type":"ContainerDied","Data":"db087feb6260250c021b57e893c12354ddf4c74fee6c40d3783a39c4117a48f7"}
Sep 11 18:45:18.013380 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:18.013351    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5faf38-wdc46"
Sep 11 18:45:18.062416 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:18.062373    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f67854-2a93-4428-a8b0-7534285b7b35-maas-api-service-ca\") pod \"84f67854-2a93-4428-a8b0-7534285b7b35\" (UID: \"84f67854-2a93-4428-a8b0-7534285b7b35\") "
Sep 11 18:45:18.062771 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:18.062747    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f67854-2a93-4428-a8b0-7534285b7b35-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "84f67854-2a93-4428-a8b0-7534285b7b35" (UID: "84f67854-2a93-4428-a8b0-7534285b7b35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:45:18.163713 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:18.163612    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f67854-2a93-4428-a8b0-7534285b7b35-maas-api-service-ca\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:45:18.882257 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:18.882225    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5faf38-wdc46"
Sep 11 18:45:18.882431 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:18.882270    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5faf38-wdc46" event={"ID":"84f67854-2a93-4428-a8b0-7534285b7b35","Type":"ContainerDied","Data":"369b116e94d7b144af7e95379ad95cca89d1dd8c3ee22c5dfbaf1b0a9a9b957f"}
Sep 11 18:45:18.882431 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:18.882313    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="369b116e94d7b144af7e95379ad95cca89d1dd8c3ee22c5dfbaf1b0a9a9b957f"
Sep 11 18:45:25.574644 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.574603    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5faf38-29819205-xpghn"]
Sep 11 18:45:25.576980 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.576949    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5faf38-29819205-xpghn"]
Sep 11 18:45:25.599690 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.599654    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4"]
Sep 11 18:45:25.599895 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.599870    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4" podUID="c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5" containerName="payload-processing" containerID="cri-o://f9b63828894d93ab8f20d85c42d2d399b9f28a52c04bf6049d617acb88ae9b2a" gracePeriod=30
Sep 11 18:45:25.610843 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.610043    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296"]
Sep 11 18:45:25.610843 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.610297    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296" podUID="69daf756-4c93-467f-ad33-4b7ed7caa74a" containerName="payload-pre-processing" containerID="cri-o://c6b359c8942b669aa1f6ff3258b6e1e68185ecaef3a7cd3c4adf570c431194e1" gracePeriod=30
Sep 11 18:45:25.774597 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.774401    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5faf38-wdc46"]
Sep 11 18:45:25.776208 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.776175    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5faf38-wdc46"]
Sep 11 18:45:25.883845 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.883806    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4"
Sep 11 18:45:25.886644 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.886620    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296"
Sep 11 18:45:25.911819 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.911784    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84f67854-2a93-4428-a8b0-7534285b7b35" path="/var/lib/kubelet/pods/84f67854-2a93-4428-a8b0-7534285b7b35/volumes"
Sep 11 18:45:25.912226 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.912206    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b1d9dff-d29f-483e-a140-44f2692080ce" path="/var/lib/kubelet/pods/8b1d9dff-d29f-483e-a140-44f2692080ce/volumes"
Sep 11 18:45:25.921376 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.921349    2576 generic.go:356] "Generic (PLEG): container finished" podID="c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5" containerID="f9b63828894d93ab8f20d85c42d2d399b9f28a52c04bf6049d617acb88ae9b2a" exitCode=0
Sep 11 18:45:25.921564 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.921392    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4"
Sep 11 18:45:25.921564 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.921439    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4" event={"ID":"c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5","Type":"ContainerDied","Data":"f9b63828894d93ab8f20d85c42d2d399b9f28a52c04bf6049d617acb88ae9b2a"}
Sep 11 18:45:25.921564 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.921508    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4" event={"ID":"c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5","Type":"ContainerDied","Data":"0101a2de7edcd01dc89d227fdc30cbcc2c84ecd82b3bf0835900de160e33986c"}
Sep 11 18:45:25.921564 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.921526    2576 scope.go:117] "RemoveContainer" containerID="f9b63828894d93ab8f20d85c42d2d399b9f28a52c04bf6049d617acb88ae9b2a"
Sep 11 18:45:25.922842 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.922819    2576 generic.go:356] "Generic (PLEG): container finished" podID="69daf756-4c93-467f-ad33-4b7ed7caa74a" containerID="c6b359c8942b669aa1f6ff3258b6e1e68185ecaef3a7cd3c4adf570c431194e1" exitCode=0
Sep 11 18:45:25.922959 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.922856    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296"
Sep 11 18:45:25.922959 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.922900    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296" event={"ID":"69daf756-4c93-467f-ad33-4b7ed7caa74a","Type":"ContainerDied","Data":"c6b359c8942b669aa1f6ff3258b6e1e68185ecaef3a7cd3c4adf570c431194e1"}
Sep 11 18:45:25.922959 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.922949    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296" event={"ID":"69daf756-4c93-467f-ad33-4b7ed7caa74a","Type":"ContainerDied","Data":"47ac10202f6e9e83c4d514f44b8e630be3376d5b5fe4b210c0d242114fc1dc88"}
Sep 11 18:45:25.931194 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.931166    2576 scope.go:117] "RemoveContainer" containerID="f9b63828894d93ab8f20d85c42d2d399b9f28a52c04bf6049d617acb88ae9b2a"
Sep 11 18:45:25.931569 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:45:25.931546    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b63828894d93ab8f20d85c42d2d399b9f28a52c04bf6049d617acb88ae9b2a\": container with ID starting with f9b63828894d93ab8f20d85c42d2d399b9f28a52c04bf6049d617acb88ae9b2a not found: ID does not exist" containerID="f9b63828894d93ab8f20d85c42d2d399b9f28a52c04bf6049d617acb88ae9b2a"
Sep 11 18:45:25.931650 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.931606    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b63828894d93ab8f20d85c42d2d399b9f28a52c04bf6049d617acb88ae9b2a"} err="failed to get container status \"f9b63828894d93ab8f20d85c42d2d399b9f28a52c04bf6049d617acb88ae9b2a\": rpc error: code = NotFound desc = could not find container \"f9b63828894d93ab8f20d85c42d2d399b9f28a52c04bf6049d617acb88ae9b2a\": container with ID starting with f9b63828894d93ab8f20d85c42d2d399b9f28a52c04bf6049d617acb88ae9b2a not found: ID does not exist"
Sep 11 18:45:25.931650 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.931630    2576 scope.go:117] "RemoveContainer" containerID="c6b359c8942b669aa1f6ff3258b6e1e68185ecaef3a7cd3c4adf570c431194e1"
Sep 11 18:45:25.939879 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.939857    2576 scope.go:117] "RemoveContainer" containerID="c6b359c8942b669aa1f6ff3258b6e1e68185ecaef3a7cd3c4adf570c431194e1"
Sep 11 18:45:25.940264 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:45:25.940237    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6b359c8942b669aa1f6ff3258b6e1e68185ecaef3a7cd3c4adf570c431194e1\": container with ID starting with c6b359c8942b669aa1f6ff3258b6e1e68185ecaef3a7cd3c4adf570c431194e1 not found: ID does not exist" containerID="c6b359c8942b669aa1f6ff3258b6e1e68185ecaef3a7cd3c4adf570c431194e1"
Sep 11 18:45:25.940349 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:25.940276    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6b359c8942b669aa1f6ff3258b6e1e68185ecaef3a7cd3c4adf570c431194e1"} err="failed to get container status \"c6b359c8942b669aa1f6ff3258b6e1e68185ecaef3a7cd3c4adf570c431194e1\": rpc error: code = NotFound desc = could not find container \"c6b359c8942b669aa1f6ff3258b6e1e68185ecaef3a7cd3c4adf570c431194e1\": container with ID starting with c6b359c8942b669aa1f6ff3258b6e1e68185ecaef3a7cd3c4adf570c431194e1 not found: ID does not exist"
Sep 11 18:45:26.032884 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:26.032847    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69daf756-4c93-467f-ad33-4b7ed7caa74a-plugins-config-volume\") pod \"69daf756-4c93-467f-ad33-4b7ed7caa74a\" (UID: \"69daf756-4c93-467f-ad33-4b7ed7caa74a\") "
Sep 11 18:45:26.033094 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:26.032897    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5-plugins-config-volume\") pod \"c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5\" (UID: \"c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5\") "
Sep 11 18:45:26.033094 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:26.032933    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6fkd\" (UniqueName: \"kubernetes.io/projected/c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5-kube-api-access-j6fkd\") pod \"c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5\" (UID: \"c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5\") "
Sep 11 18:45:26.033094 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:26.032950    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnf6s\" (UniqueName: \"kubernetes.io/projected/69daf756-4c93-467f-ad33-4b7ed7caa74a-kube-api-access-fnf6s\") pod \"69daf756-4c93-467f-ad33-4b7ed7caa74a\" (UID: \"69daf756-4c93-467f-ad33-4b7ed7caa74a\") "
Sep 11 18:45:26.033348 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:26.033322    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69daf756-4c93-467f-ad33-4b7ed7caa74a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "69daf756-4c93-467f-ad33-4b7ed7caa74a" (UID: "69daf756-4c93-467f-ad33-4b7ed7caa74a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:45:26.033420 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:26.033328    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5" (UID: "c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:45:26.035134 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:26.035105    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5-kube-api-access-j6fkd" (OuterVolumeSpecName: "kube-api-access-j6fkd") pod "c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5" (UID: "c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5"). InnerVolumeSpecName "kube-api-access-j6fkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:45:26.035262 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:26.035212    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69daf756-4c93-467f-ad33-4b7ed7caa74a-kube-api-access-fnf6s" (OuterVolumeSpecName: "kube-api-access-fnf6s") pod "69daf756-4c93-467f-ad33-4b7ed7caa74a" (UID: "69daf756-4c93-467f-ad33-4b7ed7caa74a"). InnerVolumeSpecName "kube-api-access-fnf6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:45:26.133525 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:26.133424    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6fkd\" (UniqueName: \"kubernetes.io/projected/c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5-kube-api-access-j6fkd\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:45:26.133525 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:26.133462    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnf6s\" (UniqueName: \"kubernetes.io/projected/69daf756-4c93-467f-ad33-4b7ed7caa74a-kube-api-access-fnf6s\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:45:26.133525 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:26.133471    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69daf756-4c93-467f-ad33-4b7ed7caa74a-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:45:26.133525 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:26.133480    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5-plugins-config-volume\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:45:26.244677 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:26.244635    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296"]
Sep 11 18:45:26.248456 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:26.248426    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5faf38-688fb5d5f8-xl296"]
Sep 11 18:45:26.263916 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:26.263877    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4"]
Sep 11 18:45:26.266029 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:26.265998    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5faf38-577b695bf5-mhcw4"]
Sep 11 18:45:27.909953 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:27.909917    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69daf756-4c93-467f-ad33-4b7ed7caa74a" path="/var/lib/kubelet/pods/69daf756-4c93-467f-ad33-4b7ed7caa74a/volumes"
Sep 11 18:45:27.910337 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:27.910281    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5" path="/var/lib/kubelet/pods/c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5/volumes"
Sep 11 18:45:28.213198 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:28.213161    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:45:36.212882 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:45:36.212843    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:46:56.222265 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:46:56.222224    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:47:21.621322 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:47:21.621283    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:47:29.600788 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:47:29.600712    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:47:37.819163 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:47:37.819124    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:47:53.613455 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:47:53.613418    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:47:59.605499 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:47:59.605460    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:49:29.407278 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:29.407237    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:49:35.010053 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:35.010015    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:49:45.496977 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.496937    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"]
Sep 11 18:49:45.497449 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.497428    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5" containerName="payload-processing"
Sep 11 18:49:45.497498 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.497454    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5" containerName="payload-processing"
Sep 11 18:49:45.497498 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.497470    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84f67854-2a93-4428-a8b0-7534285b7b35" containerName="revoke-keys"
Sep 11 18:49:45.497498 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.497479    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="84f67854-2a93-4428-a8b0-7534285b7b35" containerName="revoke-keys"
Sep 11 18:49:45.497722 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.497497    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69daf756-4c93-467f-ad33-4b7ed7caa74a" containerName="payload-pre-processing"
Sep 11 18:49:45.497722 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.497507    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="69daf756-4c93-467f-ad33-4b7ed7caa74a" containerName="payload-pre-processing"
Sep 11 18:49:45.497722 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.497611    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="69daf756-4c93-467f-ad33-4b7ed7caa74a" containerName="payload-pre-processing"
Sep 11 18:49:45.497722 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.497626    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="84f67854-2a93-4428-a8b0-7534285b7b35" containerName="revoke-keys"
Sep 11 18:49:45.497722 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.497637    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2d4490d-4b67-45ef-90e9-d1fe1dd43ac5" containerName="payload-processing"
Sep 11 18:49:45.500929 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.500910    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:49:45.503273 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.503252    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-66249fdb-kserve-self-signed-certs\""
Sep 11 18:49:45.510197 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.510172    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"]
Sep 11 18:49:45.667143 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.667093    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4318f6d1-386e-4c0f-859a-f75eee039d0c-dshm\") pod \"e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q\" (UID: \"4318f6d1-386e-4c0f-859a-f75eee039d0c\") " pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:49:45.667143 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.667142    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4318f6d1-386e-4c0f-859a-f75eee039d0c-tls-certs\") pod \"e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q\" (UID: \"4318f6d1-386e-4c0f-859a-f75eee039d0c\") " pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:49:45.667423 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.667168    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4318f6d1-386e-4c0f-859a-f75eee039d0c-model-cache\") pod \"e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q\" (UID: \"4318f6d1-386e-4c0f-859a-f75eee039d0c\") " pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:49:45.667423 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.667274    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxrft\" (UniqueName: \"kubernetes.io/projected/4318f6d1-386e-4c0f-859a-f75eee039d0c-kube-api-access-xxrft\") pod \"e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q\" (UID: \"4318f6d1-386e-4c0f-859a-f75eee039d0c\") " pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:49:45.667423 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.667324    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4318f6d1-386e-4c0f-859a-f75eee039d0c-home\") pod \"e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q\" (UID: \"4318f6d1-386e-4c0f-859a-f75eee039d0c\") " pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:49:45.667423 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.667397    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4318f6d1-386e-4c0f-859a-f75eee039d0c-tmp-dir\") pod \"e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q\" (UID: \"4318f6d1-386e-4c0f-859a-f75eee039d0c\") " pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:49:45.768452 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.768364    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxrft\" (UniqueName: \"kubernetes.io/projected/4318f6d1-386e-4c0f-859a-f75eee039d0c-kube-api-access-xxrft\") pod \"e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q\" (UID: \"4318f6d1-386e-4c0f-859a-f75eee039d0c\") " pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:49:45.768452 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.768407    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4318f6d1-386e-4c0f-859a-f75eee039d0c-home\") pod \"e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q\" (UID: \"4318f6d1-386e-4c0f-859a-f75eee039d0c\") " pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:49:45.768452 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.768449    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4318f6d1-386e-4c0f-859a-f75eee039d0c-tmp-dir\") pod \"e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q\" (UID: \"4318f6d1-386e-4c0f-859a-f75eee039d0c\") " pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:49:45.768753 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.768465    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4318f6d1-386e-4c0f-859a-f75eee039d0c-dshm\") pod \"e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q\" (UID: \"4318f6d1-386e-4c0f-859a-f75eee039d0c\") " pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:49:45.768753 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.768481    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4318f6d1-386e-4c0f-859a-f75eee039d0c-tls-certs\") pod \"e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q\" (UID: \"4318f6d1-386e-4c0f-859a-f75eee039d0c\") " pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:49:45.768753 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.768497    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4318f6d1-386e-4c0f-859a-f75eee039d0c-model-cache\") pod \"e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q\" (UID: \"4318f6d1-386e-4c0f-859a-f75eee039d0c\") " pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:49:45.768970 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.768944    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4318f6d1-386e-4c0f-859a-f75eee039d0c-home\") pod \"e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q\" (UID: \"4318f6d1-386e-4c0f-859a-f75eee039d0c\") " pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:49:45.769116 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.769002    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4318f6d1-386e-4c0f-859a-f75eee039d0c-tmp-dir\") pod \"e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q\" (UID: \"4318f6d1-386e-4c0f-859a-f75eee039d0c\") " pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:49:45.769116 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.769023    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4318f6d1-386e-4c0f-859a-f75eee039d0c-model-cache\") pod \"e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q\" (UID: \"4318f6d1-386e-4c0f-859a-f75eee039d0c\") " pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:49:45.770837 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.770810    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4318f6d1-386e-4c0f-859a-f75eee039d0c-dshm\") pod \"e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q\" (UID: \"4318f6d1-386e-4c0f-859a-f75eee039d0c\") " pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:49:45.770996 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.770979    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4318f6d1-386e-4c0f-859a-f75eee039d0c-tls-certs\") pod \"e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q\" (UID: \"4318f6d1-386e-4c0f-859a-f75eee039d0c\") " pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:49:45.780406 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.780375    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxrft\" (UniqueName: \"kubernetes.io/projected/4318f6d1-386e-4c0f-859a-f75eee039d0c-kube-api-access-xxrft\") pod \"e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q\" (UID: \"4318f6d1-386e-4c0f-859a-f75eee039d0c\") " pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:49:45.811719 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:45.811676    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:49:46.086779 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:46.086748    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"]
Sep 11 18:49:46.089153 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:49:46.089128    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4318f6d1_386e_4c0f_859a_f75eee039d0c.slice/crio-d7e72eebed64068e0e09333e4119287172a7faf178e49bfc0b15ea95200424a2 WatchSource:0}: Error finding container d7e72eebed64068e0e09333e4119287172a7faf178e49bfc0b15ea95200424a2: Status 404 returned error can't find the container with id d7e72eebed64068e0e09333e4119287172a7faf178e49bfc0b15ea95200424a2
Sep 11 18:49:46.104190 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:46.104165    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q" event={"ID":"4318f6d1-386e-4c0f-859a-f75eee039d0c","Type":"ContainerStarted","Data":"d7e72eebed64068e0e09333e4119287172a7faf178e49bfc0b15ea95200424a2"}
Sep 11 18:49:47.110462 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:47.110420    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q" event={"ID":"4318f6d1-386e-4c0f-859a-f75eee039d0c","Type":"ContainerStarted","Data":"afcefaa0a9924df9bba5b8c2dfc89996d3e5361e03ed0aa80190131f187cfa06"}
Sep 11 18:49:47.130750 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:47.130698    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q" podStartSLOduration=2.130682216 podStartE2EDuration="2.130682216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:49:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:47.129859431 +0000 UTC m=+2385.731220607" watchObservedRunningTime="2026-09-11 18:49:47.130682216 +0000 UTC m=+2385.732043379"
Sep 11 18:49:47.811013 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:47.810975    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:49:51.890660 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:51.890615    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"]
Sep 11 18:49:51.893878 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:51.893857    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:49:51.896324 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:51.896299    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-66249fdb-kserve-self-signed-certs\""
Sep 11 18:49:51.910041 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:51.910013    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"]
Sep 11 18:49:52.025211 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:52.025175    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxs5l\" (UniqueName: \"kubernetes.io/projected/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-kube-api-access-qxs5l\") pod \"e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l\" (UID: \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\") " pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:49:52.025427 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:52.025231    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-tmp-dir\") pod \"e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l\" (UID: \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\") " pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:49:52.025427 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:52.025260    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-home\") pod \"e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l\" (UID: \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\") " pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:49:52.025427 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:52.025323    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-model-cache\") pod \"e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l\" (UID: \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\") " pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:49:52.025427 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:52.025382    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-tls-certs\") pod \"e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l\" (UID: \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\") " pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:49:52.025427 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:52.025405    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-dshm\") pod \"e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l\" (UID: \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\") " pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:49:52.126476 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:52.126425    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-model-cache\") pod \"e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l\" (UID: \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\") " pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:49:52.126476 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:52.126483    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-tls-certs\") pod \"e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l\" (UID: \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\") " pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:49:52.126787 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:52.126633    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-dshm\") pod \"e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l\" (UID: \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\") " pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:49:52.126787 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:52.126745    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxs5l\" (UniqueName: \"kubernetes.io/projected/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-kube-api-access-qxs5l\") pod \"e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l\" (UID: \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\") " pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:49:52.126903 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:52.126793    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-tmp-dir\") pod \"e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l\" (UID: \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\") " pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:49:52.126903 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:52.126824    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-home\") pod \"e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l\" (UID: \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\") " pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:49:52.126903 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:52.126867    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-model-cache\") pod \"e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l\" (UID: \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\") " pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:49:52.127131 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:52.127108    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-tmp-dir\") pod \"e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l\" (UID: \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\") " pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:49:52.127235 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:52.127209    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-home\") pod \"e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l\" (UID: \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\") " pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:49:52.128899 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:52.128873    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-dshm\") pod \"e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l\" (UID: \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\") " pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:49:52.129110 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:52.129092    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-tls-certs\") pod \"e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l\" (UID: \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\") " pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:49:52.137594 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:52.137544    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxs5l\" (UniqueName: \"kubernetes.io/projected/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-kube-api-access-qxs5l\") pod \"e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l\" (UID: \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\") " pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:49:52.203609 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:52.203534    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:49:52.478432 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:52.478306    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"]
Sep 11 18:49:52.481438 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:49:52.481406    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfeffec18_1eaa_4b97_b4a7_434a6cd15d4c.slice/crio-72319d9416526290fb2e3753c1785d528799e57eb8aac24a5782c1c063501e20 WatchSource:0}: Error finding container 72319d9416526290fb2e3753c1785d528799e57eb8aac24a5782c1c063501e20: Status 404 returned error can't find the container with id 72319d9416526290fb2e3753c1785d528799e57eb8aac24a5782c1c063501e20
Sep 11 18:49:53.141715 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:53.141674    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l" event={"ID":"feffec18-1eaa-4b97-b4a7-434a6cd15d4c","Type":"ContainerStarted","Data":"cfd656f436187c8d59eaa0658eb8ebe3d0e28b68e8302e5b000a653504e2ca47"}
Sep 11 18:49:53.141715 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:53.141716    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l" event={"ID":"feffec18-1eaa-4b97-b4a7-434a6cd15d4c","Type":"ContainerStarted","Data":"72319d9416526290fb2e3753c1785d528799e57eb8aac24a5782c1c063501e20"}
Sep 11 18:49:53.162025 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:53.161957    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l" podStartSLOduration=2.161934238 podStartE2EDuration="2.161934238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:53.160556765 +0000 UTC m=+2391.761917931" watchObservedRunningTime="2026-09-11 18:49:53.161934238 +0000 UTC m=+2391.763295403"
Sep 11 18:49:54.711128 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:54.711088    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:49:55.811827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:55.811782    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:49:55.811827 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:55.811835    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:49:55.824269 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:55.824238    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:49:56.166833 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:49:56.166741    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:50:02.203905 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:02.203866    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:50:02.204416 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:02.203920    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:50:02.216300 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:02.216270    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:50:03.061551 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:03.061517    2576 scope.go:117] "RemoveContainer" containerID="6c97ea4c7294c45a176ef261ddf8b686425aedb17a36a701b077f7d6b48ebbb8"
Sep 11 18:50:03.069693 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:03.069671    2576 scope.go:117] "RemoveContainer" containerID="6e18d9c66dc64cad861e35a5596d15681e8d703996ff23ecb3d93e6695473573"
Sep 11 18:50:03.198828 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:03.198790    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:50:32.406466 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:32.406381    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"]
Sep 11 18:50:33.162272 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:33.162240    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"]
Sep 11 18:50:35.313727 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:35.313693    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:50:43.109435 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:43.109392    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:50:47.425625 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:47.425564    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q" podUID="4318f6d1-386e-4c0f-859a-f75eee039d0c" containerName="main" containerID="cri-o://afcefaa0a9924df9bba5b8c2dfc89996d3e5361e03ed0aa80190131f187cfa06" gracePeriod=45
Sep 11 18:50:47.669377 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:47.669350    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:50:47.786282 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:47.786243    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4318f6d1-386e-4c0f-859a-f75eee039d0c-tmp-dir\") pod \"4318f6d1-386e-4c0f-859a-f75eee039d0c\" (UID: \"4318f6d1-386e-4c0f-859a-f75eee039d0c\") "
Sep 11 18:50:47.786282 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:47.786289    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4318f6d1-386e-4c0f-859a-f75eee039d0c-dshm\") pod \"4318f6d1-386e-4c0f-859a-f75eee039d0c\" (UID: \"4318f6d1-386e-4c0f-859a-f75eee039d0c\") "
Sep 11 18:50:47.786546 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:47.786304    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4318f6d1-386e-4c0f-859a-f75eee039d0c-home\") pod \"4318f6d1-386e-4c0f-859a-f75eee039d0c\" (UID: \"4318f6d1-386e-4c0f-859a-f75eee039d0c\") "
Sep 11 18:50:47.786546 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:47.786338    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4318f6d1-386e-4c0f-859a-f75eee039d0c-tls-certs\") pod \"4318f6d1-386e-4c0f-859a-f75eee039d0c\" (UID: \"4318f6d1-386e-4c0f-859a-f75eee039d0c\") "
Sep 11 18:50:47.786546 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:47.786357    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxrft\" (UniqueName: \"kubernetes.io/projected/4318f6d1-386e-4c0f-859a-f75eee039d0c-kube-api-access-xxrft\") pod \"4318f6d1-386e-4c0f-859a-f75eee039d0c\" (UID: \"4318f6d1-386e-4c0f-859a-f75eee039d0c\") "
Sep 11 18:50:47.786546 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:47.786412    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4318f6d1-386e-4c0f-859a-f75eee039d0c-model-cache\") pod \"4318f6d1-386e-4c0f-859a-f75eee039d0c\" (UID: \"4318f6d1-386e-4c0f-859a-f75eee039d0c\") "
Sep 11 18:50:47.786794 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:47.786619    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4318f6d1-386e-4c0f-859a-f75eee039d0c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4318f6d1-386e-4c0f-859a-f75eee039d0c" (UID: "4318f6d1-386e-4c0f-859a-f75eee039d0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:47.786794 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:47.786649    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4318f6d1-386e-4c0f-859a-f75eee039d0c-home" (OuterVolumeSpecName: "home") pod "4318f6d1-386e-4c0f-859a-f75eee039d0c" (UID: "4318f6d1-386e-4c0f-859a-f75eee039d0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:47.786794 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:47.786787    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4318f6d1-386e-4c0f-859a-f75eee039d0c-tmp-dir\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:50:47.786913 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:47.786808    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4318f6d1-386e-4c0f-859a-f75eee039d0c-home\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:50:47.786913 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:47.786823    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4318f6d1-386e-4c0f-859a-f75eee039d0c-model-cache" (OuterVolumeSpecName: "model-cache") pod "4318f6d1-386e-4c0f-859a-f75eee039d0c" (UID: "4318f6d1-386e-4c0f-859a-f75eee039d0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:47.788746 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:47.788722    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4318f6d1-386e-4c0f-859a-f75eee039d0c-dshm" (OuterVolumeSpecName: "dshm") pod "4318f6d1-386e-4c0f-859a-f75eee039d0c" (UID: "4318f6d1-386e-4c0f-859a-f75eee039d0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:47.788746 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:47.788727    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4318f6d1-386e-4c0f-859a-f75eee039d0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4318f6d1-386e-4c0f-859a-f75eee039d0c" (UID: "4318f6d1-386e-4c0f-859a-f75eee039d0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:50:47.788857 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:47.788807    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4318f6d1-386e-4c0f-859a-f75eee039d0c-kube-api-access-xxrft" (OuterVolumeSpecName: "kube-api-access-xxrft") pod "4318f6d1-386e-4c0f-859a-f75eee039d0c" (UID: "4318f6d1-386e-4c0f-859a-f75eee039d0c"). InnerVolumeSpecName "kube-api-access-xxrft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:50:47.887797 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:47.887759    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4318f6d1-386e-4c0f-859a-f75eee039d0c-dshm\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:50:47.887797 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:47.887793    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4318f6d1-386e-4c0f-859a-f75eee039d0c-tls-certs\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:50:47.887797 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:47.887804    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxrft\" (UniqueName: \"kubernetes.io/projected/4318f6d1-386e-4c0f-859a-f75eee039d0c-kube-api-access-xxrft\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:50:47.888020 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:47.887814    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4318f6d1-386e-4c0f-859a-f75eee039d0c-model-cache\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:50:48.181157 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.181028    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l" podUID="feffec18-1eaa-4b97-b4a7-434a6cd15d4c" containerName="main" containerID="cri-o://cfd656f436187c8d59eaa0658eb8ebe3d0e28b68e8302e5b000a653504e2ca47" gracePeriod=45
Sep 11 18:50:48.401534 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.401500    2576 generic.go:356] "Generic (PLEG): container finished" podID="feffec18-1eaa-4b97-b4a7-434a6cd15d4c" containerID="cfd656f436187c8d59eaa0658eb8ebe3d0e28b68e8302e5b000a653504e2ca47" exitCode=0
Sep 11 18:50:48.401755 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.401570    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l" event={"ID":"feffec18-1eaa-4b97-b4a7-434a6cd15d4c","Type":"ContainerDied","Data":"cfd656f436187c8d59eaa0658eb8ebe3d0e28b68e8302e5b000a653504e2ca47"}
Sep 11 18:50:48.402949 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.402918    2576 generic.go:356] "Generic (PLEG): container finished" podID="4318f6d1-386e-4c0f-859a-f75eee039d0c" containerID="afcefaa0a9924df9bba5b8c2dfc89996d3e5361e03ed0aa80190131f187cfa06" exitCode=0
Sep 11 18:50:48.403087 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.402959    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q" event={"ID":"4318f6d1-386e-4c0f-859a-f75eee039d0c","Type":"ContainerDied","Data":"afcefaa0a9924df9bba5b8c2dfc89996d3e5361e03ed0aa80190131f187cfa06"}
Sep 11 18:50:48.403087 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.402968    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"
Sep 11 18:50:48.403087 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.402995    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q" event={"ID":"4318f6d1-386e-4c0f-859a-f75eee039d0c","Type":"ContainerDied","Data":"d7e72eebed64068e0e09333e4119287172a7faf178e49bfc0b15ea95200424a2"}
Sep 11 18:50:48.403087 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.403017    2576 scope.go:117] "RemoveContainer" containerID="afcefaa0a9924df9bba5b8c2dfc89996d3e5361e03ed0aa80190131f187cfa06"
Sep 11 18:50:48.414542 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.414514    2576 scope.go:117] "RemoveContainer" containerID="afcefaa0a9924df9bba5b8c2dfc89996d3e5361e03ed0aa80190131f187cfa06"
Sep 11 18:50:48.415177 ip-10-0-134-111 kubenswrapper[2576]: E0911 18:50:48.415155    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afcefaa0a9924df9bba5b8c2dfc89996d3e5361e03ed0aa80190131f187cfa06\": container with ID starting with afcefaa0a9924df9bba5b8c2dfc89996d3e5361e03ed0aa80190131f187cfa06 not found: ID does not exist" containerID="afcefaa0a9924df9bba5b8c2dfc89996d3e5361e03ed0aa80190131f187cfa06"
Sep 11 18:50:48.415270 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.415192    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afcefaa0a9924df9bba5b8c2dfc89996d3e5361e03ed0aa80190131f187cfa06"} err="failed to get container status \"afcefaa0a9924df9bba5b8c2dfc89996d3e5361e03ed0aa80190131f187cfa06\": rpc error: code = NotFound desc = could not find container \"afcefaa0a9924df9bba5b8c2dfc89996d3e5361e03ed0aa80190131f187cfa06\": container with ID starting with afcefaa0a9924df9bba5b8c2dfc89996d3e5361e03ed0aa80190131f187cfa06 not found: ID does not exist"
Sep 11 18:50:48.421632 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.421607    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"]
Sep 11 18:50:48.424850 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.424830    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:50:48.427412 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.427389    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-66249fdb-kserve-7656cfdbf9-qjq7q"]
Sep 11 18:50:48.494019 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.493984    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-model-cache\") pod \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\" (UID: \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\") "
Sep 11 18:50:48.494218 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.494047    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxs5l\" (UniqueName: \"kubernetes.io/projected/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-kube-api-access-qxs5l\") pod \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\" (UID: \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\") "
Sep 11 18:50:48.494218 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.494068    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-tls-certs\") pod \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\" (UID: \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\") "
Sep 11 18:50:48.494218 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.494118    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-home\") pod \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\" (UID: \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\") "
Sep 11 18:50:48.494218 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.494144    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-dshm\") pod \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\" (UID: \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\") "
Sep 11 18:50:48.494218 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.494163    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-tmp-dir\") pod \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\" (UID: \"feffec18-1eaa-4b97-b4a7-434a6cd15d4c\") "
Sep 11 18:50:48.494490 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.494322    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-model-cache" (OuterVolumeSpecName: "model-cache") pod "feffec18-1eaa-4b97-b4a7-434a6cd15d4c" (UID: "feffec18-1eaa-4b97-b4a7-434a6cd15d4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:48.494490 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.494337    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-home" (OuterVolumeSpecName: "home") pod "feffec18-1eaa-4b97-b4a7-434a6cd15d4c" (UID: "feffec18-1eaa-4b97-b4a7-434a6cd15d4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:48.494490 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.494445    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-model-cache\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:50:48.494490 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.494464    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-home\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:50:48.494718 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.494513    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "feffec18-1eaa-4b97-b4a7-434a6cd15d4c" (UID: "feffec18-1eaa-4b97-b4a7-434a6cd15d4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:48.496344 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.496320    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-kube-api-access-qxs5l" (OuterVolumeSpecName: "kube-api-access-qxs5l") pod "feffec18-1eaa-4b97-b4a7-434a6cd15d4c" (UID: "feffec18-1eaa-4b97-b4a7-434a6cd15d4c"). InnerVolumeSpecName "kube-api-access-qxs5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:50:48.496439 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.496352    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "feffec18-1eaa-4b97-b4a7-434a6cd15d4c" (UID: "feffec18-1eaa-4b97-b4a7-434a6cd15d4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:50:48.496439 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.496370    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-dshm" (OuterVolumeSpecName: "dshm") pod "feffec18-1eaa-4b97-b4a7-434a6cd15d4c" (UID: "feffec18-1eaa-4b97-b4a7-434a6cd15d4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:50:48.595250 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.595207    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxs5l\" (UniqueName: \"kubernetes.io/projected/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-kube-api-access-qxs5l\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:50:48.595250 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.595240    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-tls-certs\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:50:48.595250 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.595248    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-dshm\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:50:48.595250 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:48.595257    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feffec18-1eaa-4b97-b4a7-434a6cd15d4c-tmp-dir\") on node \"ip-10-0-134-111.ec2.internal\" DevicePath \"\""
Sep 11 18:50:49.408797 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:49.408764    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"
Sep 11 18:50:49.408985 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:49.408792    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l" event={"ID":"feffec18-1eaa-4b97-b4a7-434a6cd15d4c","Type":"ContainerDied","Data":"72319d9416526290fb2e3753c1785d528799e57eb8aac24a5782c1c063501e20"}
Sep 11 18:50:49.408985 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:49.408856    2576 scope.go:117] "RemoveContainer" containerID="cfd656f436187c8d59eaa0658eb8ebe3d0e28b68e8302e5b000a653504e2ca47"
Sep 11 18:50:49.431393 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:49.431360    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"]
Sep 11 18:50:49.435846 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:49.435812    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-66249fdb-kserve-85fff75bd7-q4d4l"]
Sep 11 18:50:49.909701 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:49.909662    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4318f6d1-386e-4c0f-859a-f75eee039d0c" path="/var/lib/kubelet/pods/4318f6d1-386e-4c0f-859a-f75eee039d0c/volumes"
Sep 11 18:50:49.910104 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:49.910090    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feffec18-1eaa-4b97-b4a7-434a6cd15d4c" path="/var/lib/kubelet/pods/feffec18-1eaa-4b97-b4a7-434a6cd15d4c/volumes"
Sep 11 18:50:50.019131 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.019095    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:50:50.297965 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.297933    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-dc4fd4b7f-29jrh"]
Sep 11 18:50:50.298321 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.298304    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feffec18-1eaa-4b97-b4a7-434a6cd15d4c" containerName="main"
Sep 11 18:50:50.298405 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.298324    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="feffec18-1eaa-4b97-b4a7-434a6cd15d4c" containerName="main"
Sep 11 18:50:50.298405 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.298339    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4318f6d1-386e-4c0f-859a-f75eee039d0c" containerName="main"
Sep 11 18:50:50.298405 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.298347    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4318f6d1-386e-4c0f-859a-f75eee039d0c" containerName="main"
Sep 11 18:50:50.298551 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.298430    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4318f6d1-386e-4c0f-859a-f75eee039d0c" containerName="main"
Sep 11 18:50:50.298551 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.298447    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="feffec18-1eaa-4b97-b4a7-434a6cd15d4c" containerName="main"
Sep 11 18:50:50.302595 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.302558    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dc4fd4b7f-29jrh"
Sep 11 18:50:50.306270 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.306243    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 18:50:50.307133 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.307105    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 18:50:50.307254 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.307132    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bqwkv\""
Sep 11 18:50:50.307254 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.307140    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:50:50.307254 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.307178    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 18:50:50.311709 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.311683    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dc4fd4b7f-29jrh"]
Sep 11 18:50:50.411173 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.411130    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a332de6d-436b-4910-b3df-859ee2a665b2-maas-api-tls\") pod \"maas-api-dc4fd4b7f-29jrh\" (UID: \"a332de6d-436b-4910-b3df-859ee2a665b2\") " pod="odh-ai-gateway-infra/maas-api-dc4fd4b7f-29jrh"
Sep 11 18:50:50.411383 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.411181    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a332de6d-436b-4910-b3df-859ee2a665b2-openshift-service-ca\") pod \"maas-api-dc4fd4b7f-29jrh\" (UID: \"a332de6d-436b-4910-b3df-859ee2a665b2\") " pod="odh-ai-gateway-infra/maas-api-dc4fd4b7f-29jrh"
Sep 11 18:50:50.411383 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.411215    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a332de6d-436b-4910-b3df-859ee2a665b2-metrics-certs\") pod \"maas-api-dc4fd4b7f-29jrh\" (UID: \"a332de6d-436b-4910-b3df-859ee2a665b2\") " pod="odh-ai-gateway-infra/maas-api-dc4fd4b7f-29jrh"
Sep 11 18:50:50.411383 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.411243    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xtc8\" (UniqueName: \"kubernetes.io/projected/a332de6d-436b-4910-b3df-859ee2a665b2-kube-api-access-7xtc8\") pod \"maas-api-dc4fd4b7f-29jrh\" (UID: \"a332de6d-436b-4910-b3df-859ee2a665b2\") " pod="odh-ai-gateway-infra/maas-api-dc4fd4b7f-29jrh"
Sep 11 18:50:50.512428 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.512386    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a332de6d-436b-4910-b3df-859ee2a665b2-maas-api-tls\") pod \"maas-api-dc4fd4b7f-29jrh\" (UID: \"a332de6d-436b-4910-b3df-859ee2a665b2\") " pod="odh-ai-gateway-infra/maas-api-dc4fd4b7f-29jrh"
Sep 11 18:50:50.512964 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.512514    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a332de6d-436b-4910-b3df-859ee2a665b2-openshift-service-ca\") pod \"maas-api-dc4fd4b7f-29jrh\" (UID: \"a332de6d-436b-4910-b3df-859ee2a665b2\") " pod="odh-ai-gateway-infra/maas-api-dc4fd4b7f-29jrh"
Sep 11 18:50:50.512964 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.512613    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a332de6d-436b-4910-b3df-859ee2a665b2-metrics-certs\") pod \"maas-api-dc4fd4b7f-29jrh\" (UID: \"a332de6d-436b-4910-b3df-859ee2a665b2\") " pod="odh-ai-gateway-infra/maas-api-dc4fd4b7f-29jrh"
Sep 11 18:50:50.512964 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.512670    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xtc8\" (UniqueName: \"kubernetes.io/projected/a332de6d-436b-4910-b3df-859ee2a665b2-kube-api-access-7xtc8\") pod \"maas-api-dc4fd4b7f-29jrh\" (UID: \"a332de6d-436b-4910-b3df-859ee2a665b2\") " pod="odh-ai-gateway-infra/maas-api-dc4fd4b7f-29jrh"
Sep 11 18:50:50.513163 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.513142    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a332de6d-436b-4910-b3df-859ee2a665b2-openshift-service-ca\") pod \"maas-api-dc4fd4b7f-29jrh\" (UID: \"a332de6d-436b-4910-b3df-859ee2a665b2\") " pod="odh-ai-gateway-infra/maas-api-dc4fd4b7f-29jrh"
Sep 11 18:50:50.514885 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.514867    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a332de6d-436b-4910-b3df-859ee2a665b2-maas-api-tls\") pod \"maas-api-dc4fd4b7f-29jrh\" (UID: \"a332de6d-436b-4910-b3df-859ee2a665b2\") " pod="odh-ai-gateway-infra/maas-api-dc4fd4b7f-29jrh"
Sep 11 18:50:50.514976 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.514959    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a332de6d-436b-4910-b3df-859ee2a665b2-metrics-certs\") pod \"maas-api-dc4fd4b7f-29jrh\" (UID: \"a332de6d-436b-4910-b3df-859ee2a665b2\") " pod="odh-ai-gateway-infra/maas-api-dc4fd4b7f-29jrh"
Sep 11 18:50:50.523432 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.523405    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xtc8\" (UniqueName: \"kubernetes.io/projected/a332de6d-436b-4910-b3df-859ee2a665b2-kube-api-access-7xtc8\") pod \"maas-api-dc4fd4b7f-29jrh\" (UID: \"a332de6d-436b-4910-b3df-859ee2a665b2\") " pod="odh-ai-gateway-infra/maas-api-dc4fd4b7f-29jrh"
Sep 11 18:50:50.615292 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.615195    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dc4fd4b7f-29jrh"
Sep 11 18:50:50.905422 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.905323    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dc4fd4b7f-29jrh"]
Sep 11 18:50:50.909843 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:50:50.909809    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda332de6d_436b_4910_b3df_859ee2a665b2.slice/crio-ae3d743740280d623d657e3b094cbb1f733c362ee2246d922ec9e4d7f6dcff86 WatchSource:0}: Error finding container ae3d743740280d623d657e3b094cbb1f733c362ee2246d922ec9e4d7f6dcff86: Status 404 returned error can't find the container with id ae3d743740280d623d657e3b094cbb1f733c362ee2246d922ec9e4d7f6dcff86
Sep 11 18:50:50.911173 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:50.911148    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:50:51.422299 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:51.422256    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dc4fd4b7f-29jrh" event={"ID":"a332de6d-436b-4910-b3df-859ee2a665b2","Type":"ContainerStarted","Data":"d2f2a04b549d68d304b3bc72ae2105e622da55fc2d09d8dc5d86653dbcb4380b"}
Sep 11 18:50:51.422483 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:51.422314    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dc4fd4b7f-29jrh" event={"ID":"a332de6d-436b-4910-b3df-859ee2a665b2","Type":"ContainerStarted","Data":"ae3d743740280d623d657e3b094cbb1f733c362ee2246d922ec9e4d7f6dcff86"}
Sep 11 18:50:51.422483 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:51.422356    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-dc4fd4b7f-29jrh"
Sep 11 18:50:51.447765 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:51.447705    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-dc4fd4b7f-29jrh" podStartSLOduration=1.071270092 podStartE2EDuration="1.447688703s" totalImagesPullingTime="376.418611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:50:51.445819414 +0000 UTC m=+2450.047180588" watchObservedRunningTime="2026-09-11 18:50:51.447688703 +0000 UTC m=+2450.049049867"
Sep 11 18:50:57.432070 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:57.432038    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-dc4fd4b7f-29jrh"
Sep 11 18:50:57.601872 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:50:57.601830    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:51:03.091431 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:03.091401    2576 scope.go:117] "RemoveContainer" containerID="192dab36e06f60498851631746d6cb2588eef903b49c43f7b5afcddb8cb60db0"
Sep 11 18:51:03.099399 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:03.099374    2576 scope.go:117] "RemoveContainer" containerID="198bfe70c2c92207d9360943d507d96c5f2225d166c2ac8244d11a92ea15e7a5"
Sep 11 18:51:04.403801 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:04.403759    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:51:10.401100 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:10.401061    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:51:17.003959 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:17.003912    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-b4kzn"]
Sep 11 18:51:25.390502 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:25.390459    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-5xzh2_80b74d21-4af5-4b9a-8164-03e70757b9fb/manager/0.log"
Sep 11 18:51:25.617332 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:25.617303    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-qvlvs_f131341e-13f0-46a6-bf5a-54c35f749073/server/0.log"
Sep 11 18:51:25.734889 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:25.734858    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rdq5p_7f99264f-7197-4667-b258-a27550fb9ad1/manager/0.log"
Sep 11 18:51:25.960723 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:25.960699    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-657fcdcb9c-5nvr6_efc58b08-02fd-4836-a9b0-9dab7ce86dec/copy-manifests/0.log"
Sep 11 18:51:25.966823 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:25.966794    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-657fcdcb9c-5nvr6_efc58b08-02fd-4836-a9b0-9dab7ce86dec/manager/0.log"
Sep 11 18:51:26.418290 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:26.418258    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-dc4fd4b7f-29jrh_a332de6d-436b-4910-b3df-859ee2a665b2/maas-api/0.log"
Sep 11 18:51:26.633504 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:26.633470    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819190-zfz2l_eb0d9e44-c87f-49b4-9d30-15be0ba26458/cleanup/0.log"
Sep 11 18:51:26.745574 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:26.745543    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819205-tvc4p_b13818ad-e848-405b-a8af-be1e9f826728/cleanup/0.log"
Sep 11 18:51:27.732466 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:27.732438    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-9szsj_2408d02c-ebba-43f0-9470-872c6ffb1ea6/manager/0.log"
Sep 11 18:51:27.971694 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:27.971667    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-nrgfh_f7d610e7-ead8-4b05-ba09-95d6f66609e1/kuadrant-console-plugin/0.log"
Sep 11 18:51:28.194813 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:28.194781    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-b4kzn_5b1c0c2f-4b2c-47e2-b0b6-cda9d763b33d/limitador/0.log"
Sep 11 18:51:29.965460 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:29.965419    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-nplmz_5cef9934-b9ad-4ce6-98b3-c268693fed89/main/0.log"
Sep 11 18:51:30.292839 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:30.292757    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-jxrxf_1648fc00-a5bf-4da2-b76b-49dd4b239b84/main/0.log"
Sep 11 18:51:38.741511 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:38.741468    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-db9kz_99a9405b-bada-4d50-a3cb-3cf9eb701eee/global-pull-secret-syncer/0.log"
Sep 11 18:51:38.915864 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:38.915828    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-j7jrb_f26c2e8d-005c-4fd3-9dc6-5cc7a8fc7232/konnectivity-agent/0.log"
Sep 11 18:51:38.957645 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:38.957613    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-111.ec2.internal_d6fbb6fad8f6855b4beb91cec165f915/haproxy/0.log"
Sep 11 18:51:43.266367 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:43.266333    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-9szsj_2408d02c-ebba-43f0-9470-872c6ffb1ea6/manager/0.log"
Sep 11 18:51:43.325089 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:43.325061    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-nrgfh_f7d610e7-ead8-4b05-ba09-95d6f66609e1/kuadrant-console-plugin/0.log"
Sep 11 18:51:43.389036 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:43.389001    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-b4kzn_5b1c0c2f-4b2c-47e2-b0b6-cda9d763b33d/limitador/0.log"
Sep 11 18:51:45.350436 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:45.350403    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f8cpd_4417b564-d430-4942-85f8-211e3098188e/node-exporter/0.log"
Sep 11 18:51:45.376314 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:45.376282    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f8cpd_4417b564-d430-4942-85f8-211e3098188e/kube-rbac-proxy/0.log"
Sep 11 18:51:45.403674 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:45.403642    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f8cpd_4417b564-d430-4942-85f8-211e3098188e/init-textfile/0.log"
Sep 11 18:51:48.887979 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:48.887945    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj"]
Sep 11 18:51:48.892670 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:48.892628    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj"
Sep 11 18:51:48.895396 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:48.895362    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8rt2b\"/\"openshift-service-ca.crt\""
Sep 11 18:51:48.895549 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:48.895376    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8rt2b\"/\"default-dockercfg-65ww9\""
Sep 11 18:51:48.896089 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:48.896069    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8rt2b\"/\"kube-root-ca.crt\""
Sep 11 18:51:48.901705 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:48.901682    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj"]
Sep 11 18:51:49.002405 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:49.002362    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe-sys\") pod \"perf-node-gather-daemonset-qg7hj\" (UID: \"4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe\") " pod="openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj"
Sep 11 18:51:49.002605 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:49.002435    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx84w\" (UniqueName: \"kubernetes.io/projected/4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe-kube-api-access-jx84w\") pod \"perf-node-gather-daemonset-qg7hj\" (UID: \"4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe\") " pod="openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj"
Sep 11 18:51:49.002605 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:49.002470    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe-proc\") pod \"perf-node-gather-daemonset-qg7hj\" (UID: \"4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe\") " pod="openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj"
Sep 11 18:51:49.002605 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:49.002516    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe-lib-modules\") pod \"perf-node-gather-daemonset-qg7hj\" (UID: \"4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe\") " pod="openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj"
Sep 11 18:51:49.002605 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:49.002540    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe-podres\") pod \"perf-node-gather-daemonset-qg7hj\" (UID: \"4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe\") " pod="openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj"
Sep 11 18:51:49.102951 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:49.102916    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe-proc\") pod \"perf-node-gather-daemonset-qg7hj\" (UID: \"4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe\") " pod="openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj"
Sep 11 18:51:49.103138 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:49.102986    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe-lib-modules\") pod \"perf-node-gather-daemonset-qg7hj\" (UID: \"4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe\") " pod="openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj"
Sep 11 18:51:49.103138 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:49.103022    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe-podres\") pod \"perf-node-gather-daemonset-qg7hj\" (UID: \"4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe\") " pod="openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj"
Sep 11 18:51:49.103138 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:49.103049    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe-proc\") pod \"perf-node-gather-daemonset-qg7hj\" (UID: \"4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe\") " pod="openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj"
Sep 11 18:51:49.103138 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:49.103053    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe-sys\") pod \"perf-node-gather-daemonset-qg7hj\" (UID: \"4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe\") " pod="openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj"
Sep 11 18:51:49.103138 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:49.103110    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx84w\" (UniqueName: \"kubernetes.io/projected/4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe-kube-api-access-jx84w\") pod \"perf-node-gather-daemonset-qg7hj\" (UID: \"4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe\") " pod="openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj"
Sep 11 18:51:49.103389 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:49.103156    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe-podres\") pod \"perf-node-gather-daemonset-qg7hj\" (UID: \"4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe\") " pod="openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj"
Sep 11 18:51:49.103389 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:49.103172    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe-lib-modules\") pod \"perf-node-gather-daemonset-qg7hj\" (UID: \"4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe\") " pod="openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj"
Sep 11 18:51:49.103389 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:49.103188    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe-sys\") pod \"perf-node-gather-daemonset-qg7hj\" (UID: \"4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe\") " pod="openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj"
Sep 11 18:51:49.111057 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:49.111024    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx84w\" (UniqueName: \"kubernetes.io/projected/4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe-kube-api-access-jx84w\") pod \"perf-node-gather-daemonset-qg7hj\" (UID: \"4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe\") " pod="openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj"
Sep 11 18:51:49.203906 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:49.203866    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj"
Sep 11 18:51:49.474644 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:49.474618    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj"]
Sep 11 18:51:49.477275 ip-10-0-134-111 kubenswrapper[2576]: W0911 18:51:49.477247    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4b279aaf_1bf1_4aa4_81f3_fbe7caab0dfe.slice/crio-edf315e1fee6f4e37743f4cc9e4d0b8cdab50054c8c962c50fcf3dfa491e7780 WatchSource:0}: Error finding container edf315e1fee6f4e37743f4cc9e4d0b8cdab50054c8c962c50fcf3dfa491e7780: Status 404 returned error can't find the container with id edf315e1fee6f4e37743f4cc9e4d0b8cdab50054c8c962c50fcf3dfa491e7780
Sep 11 18:51:49.679655 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:49.679620    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj" event={"ID":"4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe","Type":"ContainerStarted","Data":"b9be13b6fdcfd714ee81ebe93b434898819f49c10f76bec67caee00ec133c0f4"}
Sep 11 18:51:49.679829 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:49.679670    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj" event={"ID":"4b279aaf-1bf1-4aa4-81f3-fbe7caab0dfe","Type":"ContainerStarted","Data":"edf315e1fee6f4e37743f4cc9e4d0b8cdab50054c8c962c50fcf3dfa491e7780"}
Sep 11 18:51:49.679829 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:49.679755    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj"
Sep 11 18:51:49.694930 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:49.694875    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj" podStartSLOduration=1.694853788 podStartE2EDuration="1.694853788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:51:49.694559101 +0000 UTC m=+2508.295920264" watchObservedRunningTime="2026-09-11 18:51:49.694853788 +0000 UTC m=+2508.296214953"
Sep 11 18:51:51.039436 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:51.039406    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7d7v2_b5b1dd54-6a88-46eb-8f13-0e7a54c9490e/dns/0.log"
Sep 11 18:51:51.062036 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:51.062007    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7d7v2_b5b1dd54-6a88-46eb-8f13-0e7a54c9490e/kube-rbac-proxy/0.log"
Sep 11 18:51:51.203870 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:51.203840    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nqz9g_e0a9600b-c5a4-4e01-99b5-fe0598a4cb52/dns-node-resolver/0.log"
Sep 11 18:51:51.863101 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:51.863062    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ckcv2_9c70edc6-7469-4d7c-b3f1-5d88caac76aa/node-ca/0.log"
Sep 11 18:51:54.418922 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:54.418843    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xzgnx_e64c4597-e3bc-46f1-8118-ab47d1e632cb/serve-healthcheck-canary/0.log"
Sep 11 18:51:55.021866 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:55.021833    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9zc7s_f7dcfb0b-9065-4c7e-a163-73844899ac4c/kube-rbac-proxy/0.log"
Sep 11 18:51:55.043043 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:55.043011    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9zc7s_f7dcfb0b-9065-4c7e-a163-73844899ac4c/exporter/0.log"
Sep 11 18:51:55.065015 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:55.064981    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9zc7s_f7dcfb0b-9065-4c7e-a163-73844899ac4c/extractor/0.log"
Sep 11 18:51:55.694667 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:55.694640    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8rt2b/perf-node-gather-daemonset-qg7hj"
Sep 11 18:51:57.944855 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:57.944818    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-5xzh2_80b74d21-4af5-4b9a-8164-03e70757b9fb/manager/0.log"
Sep 11 18:51:58.030697 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:58.030664    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-qvlvs_f131341e-13f0-46a6-bf5a-54c35f749073/server/0.log"
Sep 11 18:51:58.175073 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:58.174989    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rdq5p_7f99264f-7197-4667-b258-a27550fb9ad1/manager/0.log"
Sep 11 18:51:58.302526 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:58.302483    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-657fcdcb9c-5nvr6_efc58b08-02fd-4836-a9b0-9dab7ce86dec/manager/0.log"
Sep 11 18:51:58.324594 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:58.324555    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-657fcdcb9c-5nvr6_efc58b08-02fd-4836-a9b0-9dab7ce86dec/copy-manifests/0.log"
Sep 11 18:51:59.787151 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:59.787116    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6dd5f9dbb5-qb7nd_e0e19b41-5055-4885-9d70-df53bb3a9e87/manager/0.log"
Sep 11 18:51:59.815028 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:51:59.815002    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-fdcmk_14ba344f-31dc-4a15-882e-9ef608ece7d4/openshift-lws-operator/0.log"
Sep 11 18:52:03.126634 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:52:03.126604    2576 scope.go:117] "RemoveContainer" containerID="db087feb6260250c021b57e893c12354ddf4c74fee6c40d3783a39c4117a48f7"
Sep 11 18:52:06.469296 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:52:06.469272    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-759db_fe7af037-cf3e-4ae8-bdf0-f458a249950f/kube-multus-additional-cni-plugins/0.log"
Sep 11 18:52:06.493487 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:52:06.493455    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-759db_fe7af037-cf3e-4ae8-bdf0-f458a249950f/egress-router-binary-copy/0.log"
Sep 11 18:52:06.522965 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:52:06.522934    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-759db_fe7af037-cf3e-4ae8-bdf0-f458a249950f/cni-plugins/0.log"
Sep 11 18:52:06.544842 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:52:06.544812    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-759db_fe7af037-cf3e-4ae8-bdf0-f458a249950f/bond-cni-plugin/0.log"
Sep 11 18:52:06.566978 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:52:06.566939    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-759db_fe7af037-cf3e-4ae8-bdf0-f458a249950f/routeoverride-cni/0.log"
Sep 11 18:52:06.605907 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:52:06.605876    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-759db_fe7af037-cf3e-4ae8-bdf0-f458a249950f/whereabouts-cni-bincopy/0.log"
Sep 11 18:52:06.630435 ip-10-0-134-111 kubenswrapper[2576]: I0911 18:52:06.630399    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-759db_fe7af037-cf3e-4ae8-bdf0-f458a249950f/whereabouts-cni/0.log"